Aug 04 10:25:41.857326 ip-10-0-139-198 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 10:25:41.857338 ip-10-0-139-198 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 10:25:41.857346 ip-10-0-139-198 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 10:25:41.857564 ip-10-0-139-198 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 10:25:52.098635 ip-10-0-139-198 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 10:25:52.098653 ip-10-0-139-198 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 27a9d1767e4e42c092e070633730136f -- Aug 04 10:27:18.924113 ip-10-0-139-198 systemd[1]: Starting Kubernetes Kubelet... Aug 04 10:27:19.366384 ip-10-0-139-198 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:27:19.366384 ip-10-0-139-198 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 10:27:19.366384 ip-10-0-139-198 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:27:19.366384 ip-10-0-139-198 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 10:27:19.367053 ip-10-0-139-198 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 10:27:19.369262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.369179 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 10:27:19.374166 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374137 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:19.374166 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374168 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374175 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374180 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374206 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374210 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374214 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374216 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374220 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374224 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374228 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374233 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374238 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374242 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374247 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374250 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374269 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:19.374309 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374272 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374407 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374415 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374417 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374420 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374423 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374425 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374428 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374431 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374433 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374436 2577 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374438 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374441 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374443 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374445 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374448 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374450 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374453 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374455 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374457 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:19.374786 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374460 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374462 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374467 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374471 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374473 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374476 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374479 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374481 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374484 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374490 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374493 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374497 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374500 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374503 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374507 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374509 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374512 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374515 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:19.375276 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374518 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374520 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374523 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374525 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374528 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374530 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374533 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374535 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374538 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374540 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374543 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374545 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374548 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374550 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374552 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374555 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374558 2577 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374561 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374564 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374566 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:19.375699 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374568 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374571 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374574 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374576 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374581 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374585 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374588 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374590 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374595 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374597 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374600 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374603 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374605 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374608 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374610 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374613 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374615 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374618 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374621 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374623 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:19.376177 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374625 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374628 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374631 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.374633 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376241 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376249 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376252 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376255 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376259 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376261 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376264 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376266 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376269 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376272 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376275 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376278 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376282 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376286 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376289 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376292 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:19.376652 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376294 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376298 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376300 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376303 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376306 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376308 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376310 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376313 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376316 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376318 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376321 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376323 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376325 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376328 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376332 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376334 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376337 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376340 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376342 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:19.377156 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376345 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376348 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376352 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376355 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376358 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376360 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376363 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376365 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376368 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376370 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376373 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376376 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376378 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376381 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376383 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376387 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376389 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376392 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376394 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376396 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:19.377626 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376399 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376401 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376403 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376406 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376409 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376411 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376414 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376417 2577 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376420 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376422 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376425 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376427 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376430 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376432 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376435 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376437 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376440 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376442 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376445 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376447 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:19.378114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376449 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376452 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376455 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376457 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376461 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376463 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376466 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376468 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376471 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376473 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376477 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376481 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376484 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376486 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376489 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376491 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376494 2577 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376497 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376499 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:19.378609 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376503 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376505 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376508 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376511 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376514 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376886 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376898 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376901 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376904 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376906 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376909 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376912 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376914 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376916 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376919 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376921 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376924 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376926 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376929 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:19.379067 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376931 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376934 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376937 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376940 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376943 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376946 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376948 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376951 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376953 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376955 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376958 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376961 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376963 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376966 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376969 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376972 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376974 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376977 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376981 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:19.379500 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376984 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376987 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376991 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376993 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376996 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.376998 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377002 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377004 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377007 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377009 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377011 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377014 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377017 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377019 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377021 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377024 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377027 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377030 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377033 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377035 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:19.379992 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377037 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377039 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377042 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377044 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377046 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377049 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377051 2577 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377054 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377057 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377059 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377062 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377064 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377066 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377069 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377071 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377074 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377076 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377079 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377081 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377083 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:19.380451 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377086 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377090 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377093 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377096 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377098 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377100 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377103 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377105 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377108 2577 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377111 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377113 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377116 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377119 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377121 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377124 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377126 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377129 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377131 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377134 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377136 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:19.380931 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377138 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377141 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377143 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377145 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377148 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377152 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377218 2577 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377225 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377235 2577 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377239 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377243 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377246 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377250 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377254 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377257 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377260 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377263 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377267 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377269 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377272 2577 flags.go:64] FLAG: --cgroup-root="" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377275 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377278 2577 flags.go:64] FLAG: --client-ca-file="" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377282 2577 flags.go:64] FLAG: --cloud-provider="external" Aug 04 10:27:19.381434 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377285 2577 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377289 2577 flags.go:64] FLAG: --cluster-domain="" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377291 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377294 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377297 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377301 2577 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377305 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377308 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377310 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377313 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377316 2577 flags.go:64] FLAG: --contention-profiling="false" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377319 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377322 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377325 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377328 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377332 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377335 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377337 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377340 2577 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377343 2577 flags.go:64] FLAG: --enable-server="true" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377346 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377350 2577 flags.go:64] FLAG: --event-burst="100" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377353 2577 flags.go:64] FLAG: --event-qps="50" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377356 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 10:27:19.381993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377358 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377361 2577 flags.go:64] FLAG: --eviction-hard="" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377364 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377367 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377370 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377373 2577 flags.go:64] FLAG: --eviction-soft="" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377376 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377378 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377382 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377384 2577 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377387 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377390 2577 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377392 2577 flags.go:64] FLAG: --feature-gates="" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377396 2577 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377399 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377402 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377405 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377408 2577 flags.go:64] FLAG: --healthz-port="10248" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377411 2577 flags.go:64] FLAG: --help="false" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377414 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377417 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377419 2577 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377422 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377425 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 10:27:19.382612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377428 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377431 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377434 2577 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377437 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377439 2577 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377442 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377445 2577 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377447 2577 flags.go:64] FLAG: --kube-reserved="" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377450 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377453 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377456 2577 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377458 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377461 2577 flags.go:64] FLAG: --lock-file="" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377463 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377468 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377471 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377476 2577 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377480 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377482 2577 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377485 2577 flags.go:64] FLAG: --logging-format="text" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377488 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377491 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377494 2577 flags.go:64] FLAG: --manifest-url="" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377497 2577 flags.go:64] FLAG: --manifest-url-header="" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377501 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 10:27:19.383201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377503 2577 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377507 2577 flags.go:64] FLAG: --max-pods="110" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377510 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377513 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377516 2577 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377519 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377522 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377525 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377528 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377534 2577 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377537 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377540 2577 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377542 2577 flags.go:64] FLAG: --pod-cidr="" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377545 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377550 2577 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377553 2577 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377556 2577 flags.go:64] FLAG: --pods-per-core="0" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377558 2577 flags.go:64] FLAG: --port="10250" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377561 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377564 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05489bef4a69f43ed" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377567 2577 flags.go:64] FLAG: --qos-reserved="" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377570 2577 flags.go:64] FLAG: --read-only-port="10255" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377576 2577 flags.go:64] FLAG: --register-node="true" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377579 2577 flags.go:64] FLAG: --register-with-taints="" Aug 04 10:27:19.383808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377582 2577 flags.go:64] FLAG: --registry-burst="10" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377586 2577 flags.go:64] FLAG: --registry-qps="5" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377588 2577 flags.go:64] FLAG: --reserved-cpus="" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377591 2577 flags.go:64] FLAG: --reserved-memory="" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377595 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377598 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377601 2577 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377604 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377607 2577 flags.go:64] FLAG: --runonce="false" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377610 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377612 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377615 2577 flags.go:64] FLAG: --seccomp-default="false" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377618 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377621 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377624 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377627 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377630 2577 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377632 2577 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377635 2577 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377638 2577 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377641 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377644 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377647 2577 flags.go:64] FLAG: --system-cgroups="" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377650 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377655 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 10:27:19.384425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377658 2577 flags.go:64] FLAG: --tls-cert-file="" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377661 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377664 2577 flags.go:64] FLAG: --tls-min-version="" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377667 2577 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377669 2577 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377673 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377676 2577 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377679 2577 flags.go:64] FLAG: --v="2" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377683 2577 flags.go:64] FLAG: --version="false" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377687 2577 flags.go:64] FLAG: --vmodule="" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377691 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.377695 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377803 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377807 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377810 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377813 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377816 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377819 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377822 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377825 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377839 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377842 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:19.385076 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377845 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377849 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377853 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377856 2577 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377859 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377862 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377865 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377868 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377871 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377873 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377876 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377879 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377882 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377885 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377888 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377892 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377894 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377897 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377900 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377903 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:19.385590 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377906 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377910 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377913 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377916 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377919 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377921 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377924 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377926 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377929 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377931 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377934 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377936 2577 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377939 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377941 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377944 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377947 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377949 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377952 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377954 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:19.386079 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377957 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377960 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377964 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377966 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377969 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377972 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377974 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377977 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377981 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377983 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377986 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377988 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377991 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377993 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.377998 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378000 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378002 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378005 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378007 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378010 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:19.386527 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378012 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378015 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378017 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378020 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378023 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378025 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378027 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378030 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378032 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378035 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378037 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378040 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378042 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378045 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378047 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378050 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378052 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378055 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378057 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378059 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:19.387020 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378063 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378066 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378068 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378071 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378073 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378076 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378081 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378084 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378087 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.378091 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.378807 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.385622 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.385637 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.385655 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 10:27:19.387485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.385661 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385718 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385723 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385726 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385729 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385733 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385737 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385740 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385743 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385746 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385749 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385751 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385754 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385756 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385759 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385762 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385765 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385767 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385770 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385772 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:19.387822 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385775 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385777 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385780 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385782 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385785 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385787 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385790 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385793 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385796 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385798 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385802 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385805 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385808 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385810 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385813 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385816 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385818 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385821 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385823 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385826 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:19.388295 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385843 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385846 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385849 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385851 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385854 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385856 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385859 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385862 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385864 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385867 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385870 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385872 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385876 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385879 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385882 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385884 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385887 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385890 2577 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385892 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385895 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:19.388749 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385897 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385900 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385904 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385907 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385910 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385912 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385915 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385919 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385922 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385925 2577 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385928 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385931 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385934 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385937 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385939 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385942 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385945 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385947 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385950 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385953 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:19.389263 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385955 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385958 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385960 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385963 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385965 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385968 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385970 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385973 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385975 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385978 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385980 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385983 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385985 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385988 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385991 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385994 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385997 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.385999 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386002 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:19.389730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386004 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.386009 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386115 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386120 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386123 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386126 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386129 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386131 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386134 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386136 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386140 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386142 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386145 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386147 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 10:27:19.390194 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386150 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386152 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386155 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386157 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386160 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386162 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386165 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386167 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386170 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386172 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386175 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386177 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386180 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386182 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386186 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386188 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386191 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386194 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386197 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386199 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 10:27:19.390531 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386202 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386204 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386207 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386209 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386212 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386214 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386216 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386219 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386221 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386224 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386227 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386229 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386233 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386236 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386239 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386241 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386244 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386247 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386249 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386252 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 10:27:19.391107 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386254 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386258 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386261 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386264 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386267 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386270 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386273 2577 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386276 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386279 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386282 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386285 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386287 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386289 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386292 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386295 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386297 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386300 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386302 2577 feature_gate.go:328] unrecognized feature gate: Example Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386305 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386307 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 10:27:19.391568 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386310 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386313 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386315 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386318 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386320 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386322 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386325 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386327 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386330 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386332 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386335 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386337 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386339 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386342 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386344 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386347 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386349 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386352 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386356 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386359 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 10:27:19.392185 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386361 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386364 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386366 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386369 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386372 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386374 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:19.386377 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.386381 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.386472 2577 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.388777 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.390548 2577 server.go:1019] "Starting client certificate rotation" Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.390664 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 10:27:19.392710 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.390702 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 10:27:19.393331 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.393317 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 10:27:19.394413 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.394387 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 10:27:19.406457 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.406438 2577 log.go:25] "Validated CRI v1 runtime API" Aug 04 10:27:19.409040 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.409026 2577 log.go:25] "Validated CRI v1 image API" Aug 04 10:27:19.409812 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.409797 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 10:27:19.412421 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.412403 2577 fs.go:135] Filesystem UUIDs: map[4af108bf-be45-4772-b2e5-b2ce41391e4a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d5cbf7d6-e1ed-4584-bd38-ee672e0a1938:/dev/nvme0n1p4] Aug 04 10:27:19.412489 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.412419 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 10:27:19.417576 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.417470 2577 manager.go:216] Machine: {Timestamp:2026-08-04 10:27:19.416054763 +0000 UTC m=+0.367462302 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118714 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d9094451000cd4af679d9d3875f8 SystemUUID:ec28d909-4451-000c-d4af-679d9d3875f8 BootID:27a9d176-7e4e-42c0-92e0-70633730136f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:bf:e0:b9:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:bf:e0:b9:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:70:a0:3b:0b:2b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 10:27:19.417576 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.417566 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 10:27:19.417704 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.417633 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 10:27:19.421114 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.421089 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 10:27:19.421265 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.421116 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-198.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 10:27:19.422007 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.421992 2577 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 10:27:19.422074 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.422018 2577 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 10:27:19.422074 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.422029 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 10:27:19.422074 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.422054 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 10:27:19.422074 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.422064 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 10:27:19.423400 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.423382 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 10:27:19.428772 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.428761 2577 state_mem.go:36] "Initialized new in-memory state store" Aug 04 10:27:19.428914 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.428904 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 10:27:19.428953 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.428934 2577 kubelet.go:486] "Attempting to sync node with API server" Aug 04 10:27:19.428953 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.428945 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 10:27:19.429003 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.428956 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 10:27:19.429003 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.428965 2577 kubelet.go:398] "Adding apiserver pod source" Aug 04 10:27:19.429003 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.428974 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 10:27:19.429336 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.429318 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 10:27:19.429336 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.429334 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 10:27:19.431786 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.431765 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 10:27:19.433736 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.433723 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 10:27:19.433772 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.433752 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 10:27:19.435874 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435859 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435879 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435885 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435890 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435899 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435904 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435911 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435916 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435929 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 10:27:19.435936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435935 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 10:27:19.436169 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435942 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 10:27:19.436169 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.435947 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 10:27:19.436819 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.436806 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 10:27:19.436819 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.436815 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 10:27:19.439843 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.439815 2577 server.go:1268] "Started kubelet" Aug 04 10:27:19.439924 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.439891 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 10:27:19.440467 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.440178 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 10:27:19.440577 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.440566 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 10:27:19.440666 ip-10-0-139-198 systemd[1]: Started Kubernetes Kubelet. Aug 04 10:27:19.447544 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.447522 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 10:27:19.448304 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.448283 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-198.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 10:27:19.449382 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.449350 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 10:27:19.449449 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.449352 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 10:27:19.449676 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.449654 2577 server.go:310] "Adding debug handlers to kubelet server" Aug 04 10:27:19.449887 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.449773 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 10:27:19.453928 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.452781 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-198.ec2.internal.18c8935394c32153 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-198.ec2.internal,UID:ip-10-0-139-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-198.ec2.internal,},FirstTimestamp:2026-08-04 10:27:19.439786323 +0000 UTC m=+0.391193864,LastTimestamp:2026-08-04 10:27:19.439786323 +0000 UTC m=+0.391193864,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-198.ec2.internal,}" Aug 04 10:27:19.455928 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.455913 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 10:27:19.456001 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.455930 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 10:27:19.456065 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.456041 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:19.456688 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.456666 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 10:27:19.456771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.456674 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 10:27:19.456771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.456715 2577 factory.go:55] Registering systemd factory Aug 04 10:27:19.456771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.456726 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 10:27:19.456771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.456744 2577 factory.go:223] Registration of the systemd container factory successfully Aug 04 10:27:19.456995 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.456814 2577 reconstruct.go:97] "Volume reconstruction finished" Aug 04 10:27:19.456995 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.456823 2577 reconciler.go:29] "Reconciler: start to sync state" Aug 04 10:27:19.457096 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.457025 2577 factory.go:153] Registering CRI-O factory Aug 04 10:27:19.457096 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.457041 2577 factory.go:223] Registration of the crio container factory successfully Aug 04 10:27:19.457096 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.457090 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 10:27:19.457354 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.457114 2577 factory.go:103] Registering Raw factory Aug 04 10:27:19.457354 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.457129 2577 manager.go:1196] Started watching for new ooms in manager Aug 04 10:27:19.457630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.457572 2577 manager.go:349] Starting recovery of all containers Aug 04 10:27:19.463459 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.463424 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 10:27:19.467648 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.467630 2577 manager.go:354] Recovery completed Aug 04 10:27:19.469420 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.469389 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Aug 04 10:27:19.472161 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.472149 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:19.472548 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.472531 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 10:27:19.472621 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.472587 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-198.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 10:27:19.474506 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.474490 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:19.474570 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.474515 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:19.474570 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.474526 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:19.475092 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.475048 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 10:27:19.475092 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.475060 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 10:27:19.475092 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.475076 2577 state_mem.go:36] "Initialized new in-memory state store" Aug 04 10:27:19.475247 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.475060 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:19.475910 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.475893 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:19.475975 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.475948 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:19.475975 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.475963 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:19.477345 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.477333 2577 policy_none.go:49] "None policy: Start" Aug 04 10:27:19.477405 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.477348 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 10:27:19.477405 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.477358 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 10:27:19.479349 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.479334 2577 policy_none.go:47] "Start" Aug 04 10:27:19.485134 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.485072 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-198.ec2.internal.18c8935396d4e05b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-198.ec2.internal,UID:ip-10-0-139-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-198.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-198.ec2.internal,},FirstTimestamp:2026-08-04 10:27:19.474503771 +0000 UTC m=+0.425911313,LastTimestamp:2026-08-04 10:27:19.474503771 +0000 UTC m=+0.425911313,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-198.ec2.internal,}" Aug 04 10:27:19.493869 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.493846 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jn2r6" Aug 04 10:27:19.500666 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.500609 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-198.ec2.internal.18c8935396d51ebe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-198.ec2.internal,UID:ip-10-0-139-198.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-198.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-198.ec2.internal,},FirstTimestamp:2026-08-04 10:27:19.474519742 +0000 UTC m=+0.425927282,LastTimestamp:2026-08-04 10:27:19.474519742 +0000 UTC m=+0.425927282,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-198.ec2.internal,}" Aug 04 10:27:19.505659 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.505642 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jn2r6" Aug 04 10:27:19.512605 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.512589 2577 manager.go:337] "Starting Device Plugin manager" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.512628 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.512641 2577 server.go:87] "Starting device plugin registration server" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.512900 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.512912 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.513025 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.513093 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.513102 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.513488 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.513533 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.525591 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.525624 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.525640 2577 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.525666 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 10:27:19.536823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.528701 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:19.613965 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.613936 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:19.614839 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.614815 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:19.614965 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.614860 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:19.614965 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.614872 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:19.614965 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.614895 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.626450 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.626396 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal"] Aug 04 10:27:19.626516 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.626459 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:19.627223 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.627209 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:19.627303 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.627236 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:19.627303 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.627246 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:19.627892 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.627877 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.627970 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.627897 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-198.ec2.internal\": node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:19.629573 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.629561 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:19.629707 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.629681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.629787 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.629719 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:19.630239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.630225 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:19.630316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.630247 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:19.630316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.630258 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:19.630316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.630273 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:19.630316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.630294 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:19.630316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.630305 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:19.632582 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.632567 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.632651 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.632593 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:19.633182 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.633167 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:19.633250 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.633192 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:19.633250 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.633205 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:19.658383 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.658236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de1bfd90085cee16926246b78cba2cc2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal\" (UID: \"de1bfd90085cee16926246b78cba2cc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.658452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.658397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de1bfd90085cee16926246b78cba2cc2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal\" (UID: \"de1bfd90085cee16926246b78cba2cc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.658452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.658418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5230571842e399889e9775bf9720e82e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-198.ec2.internal\" (UID: \"5230571842e399889e9775bf9720e82e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.661387 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.661371 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-198.ec2.internal\" not found" node="ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.664672 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.664658 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-198.ec2.internal\" not found" node="ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.666888 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.666875 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:19.758956 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.758931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de1bfd90085cee16926246b78cba2cc2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal\" (UID: \"de1bfd90085cee16926246b78cba2cc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.758956 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.758960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de1bfd90085cee16926246b78cba2cc2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal\" (UID: \"de1bfd90085cee16926246b78cba2cc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.759135 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.758980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5230571842e399889e9775bf9720e82e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-198.ec2.internal\" (UID: \"5230571842e399889e9775bf9720e82e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.759135 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.759020 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5230571842e399889e9775bf9720e82e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-198.ec2.internal\" (UID: \"5230571842e399889e9775bf9720e82e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.759135 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.759032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de1bfd90085cee16926246b78cba2cc2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal\" (UID: \"de1bfd90085cee16926246b78cba2cc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.759135 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.759041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de1bfd90085cee16926246b78cba2cc2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal\" (UID: \"de1bfd90085cee16926246b78cba2cc2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.767320 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.767300 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:19.867960 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.867927 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:19.964860 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.964780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:19.968919 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:19.968900 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:19.968975 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:19.968943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" Aug 04 10:27:20.069779 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.069741 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.170205 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.170179 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.270777 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.270743 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.340854 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.340820 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:20.371796 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.371774 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.389891 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.389873 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 10:27:20.390020 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.390002 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 10:27:20.390059 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.390026 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 10:27:20.472704 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.472680 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.511510 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.511481 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 10:22:19 +0000 UTC" deadline="2028-02-06 10:13:20.841082745 +0000 UTC" Aug 04 10:27:20.511510 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.511510 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13223h46m0.329576123s" Aug 04 10:27:20.569859 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:20.569816 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5230571842e399889e9775bf9720e82e.slice/crio-a23734ddd4dddd7d79ab5c359b860100e02521521e639a70ad23bc5febf857d9 WatchSource:0}: Error finding container a23734ddd4dddd7d79ab5c359b860100e02521521e639a70ad23bc5febf857d9: Status 404 returned error can't find the container with id a23734ddd4dddd7d79ab5c359b860100e02521521e639a70ad23bc5febf857d9 Aug 04 10:27:20.570559 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:20.570534 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1bfd90085cee16926246b78cba2cc2.slice/crio-f59528aa6da1daf14972f960961970bb864430b7435040dc112ff7ca037c1866 WatchSource:0}: Error finding container f59528aa6da1daf14972f960961970bb864430b7435040dc112ff7ca037c1866: Status 404 returned error can't find the container with id f59528aa6da1daf14972f960961970bb864430b7435040dc112ff7ca037c1866 Aug 04 10:27:20.573465 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.573446 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.575591 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.575579 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:27:20.664170 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.664145 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 10:27:20.665259 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.665238 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientMemory" Aug 04 10:27:20.665344 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.665270 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 10:27:20.665344 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:20.665282 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeHasSufficientPID" Aug 04 10:27:20.673884 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.673856 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.774681 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.774660 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.875058 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.874999 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:20.975331 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:20.975301 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-198.ec2.internal\" not found" Aug 04 10:27:21.045128 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.045107 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:21.050338 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.050321 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:21.056517 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.056499 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" Aug 04 10:27:21.073823 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.073807 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 10:27:21.074959 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.074947 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" Aug 04 10:27:21.084087 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.084072 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 10:27:21.216446 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.216392 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:21.430139 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.430108 2577 apiserver.go:52] "Watching apiserver" Aug 04 10:27:21.438154 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.437768 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 10:27:21.440354 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.440001 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-wdm6f","kube-system/konnectivity-agent-6h49x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l","openshift-cluster-node-tuning-operator/tuned-5bx2n","openshift-dns/node-resolver-q4mkf","openshift-multus/network-metrics-daemon-xrhhd","openshift-network-diagnostics/network-check-target-5gw46","openshift-network-operator/iptables-alerter-bt7sr","kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal","openshift-image-registry/node-ca-bznw4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal","openshift-multus/multus-8jftd","openshift-multus/multus-additional-cni-plugins-p82pr","openshift-ovn-kubernetes/ovnkube-node-clkh9"] Aug 04 10:27:21.443203 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.442789 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:21.443203 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.442861 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:21.445659 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.444999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.447152 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.447015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.450094 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.449577 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.452939 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.452023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.454334 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.454104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:21.454334 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.454166 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:21.455166 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455140 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 10:27:21.455290 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455268 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 10:27:21.455462 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455444 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sw94l\"" Aug 04 10:27:21.455596 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455553 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 10:27:21.455676 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455618 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 10:27:21.455676 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455657 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pbb8t\"" Aug 04 10:27:21.455784 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455451 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 10:27:21.455868 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455817 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tdbds\"" Aug 04 10:27:21.455932 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455861 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 10:27:21.455932 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.455908 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 10:27:21.456040 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.456007 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 10:27:21.456107 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.456090 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:27:21.457040 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.457021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.457867 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.457844 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cjz64\"" Aug 04 10:27:21.458089 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.458071 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 10:27:21.458188 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.458169 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 10:27:21.458295 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.458263 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 10:27:21.458356 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.458349 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 10:27:21.459192 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.459174 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.461671 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.461648 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 10:27:21.462007 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.461989 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 10:27:21.462093 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.462043 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 10:27:21.462301 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.462258 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:27:21.462433 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.462406 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 10:27:21.462606 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.462586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.462700 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.462660 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64vqm\"" Aug 04 10:27:21.462850 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.462834 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 10:27:21.463907 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.463887 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ffgmx\"" Aug 04 10:27:21.464094 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.464079 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 10:27:21.464319 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.464301 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 10:27:21.464385 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.464371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.465246 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.465228 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r7bpb\"" Aug 04 10:27:21.465632 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.465614 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 10:27:21.466439 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.466420 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 10:27:21.466763 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.466721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.469275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.469275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-socket-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.469275 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.469227 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:21.469275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-cni-netd\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:21.469528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-modprobe-d\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.469528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-slash\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.469528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469432 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-sys-fs\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.469528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-run-ovn-kubernetes\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-cni-bin\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-run-netns\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-ovn\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-env-overrides\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27psm\" (UniqueName: \"kubernetes.io/projected/4bce4a74-6633-4b34-8d8d-86c76d30dcce-kube-api-access-27psm\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef06db54-e625-4946-bf56-7be5c68f9a94-tmp\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x25kx\" (UniqueName: \"kubernetes.io/projected/642e5a13-95f9-42bd-a40e-d74063aa8d4b-kube-api-access-x25kx\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469843 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 10:27:21.469841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5sdj\" (UniqueName: \"kubernetes.io/projected/0fa238f6-a750-4b82-8c0c-d31e0fcda574-kube-api-access-k5sdj\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-system-cni-dir\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-cni-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-os-release\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.469980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysconfig\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-tuned\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-device-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-etc-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470118 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2d8gs\"" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-daemon-config\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-run\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-cni-bin\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470171 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/334b78c8-0fb9-46b2-942d-b634710b26b7-host-slash\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470169 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-conf-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-registration-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.470372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0416d84-cde5-4c8a-a141-40796d09738b-ovn-node-metrics-cert\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470241 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-ovnkube-script-lib\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-cnibin\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bce4a74-6633-4b34-8d8d-86c76d30dcce-cni-binary-copy\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-netns\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-kubelet\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470470 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-ovnkube-config\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-hostroot\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-cni-binary-copy\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-cni-multus\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysctl-d\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470623 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-systemd\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-lib-modules\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8n2t\" (UniqueName: \"kubernetes.io/projected/0f593643-7d48-4bfd-b9c4-1ba164f23119-kube-api-access-v8n2t\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470726 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-kubernetes\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-systemd\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-node-log\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-cnibin\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-kubelet\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-k8s-cni-cncf-io\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/334b78c8-0fb9-46b2-942d-b634710b26b7-iptables-alerter-script\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-var-lib-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.470994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jv4\" (UniqueName: \"kubernetes.io/projected/c0416d84-cde5-4c8a-a141-40796d09738b-kube-api-access-72jv4\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471024 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-system-cni-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysctl-conf\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-host\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpgv\" (UniqueName: \"kubernetes.io/projected/ef06db54-e625-4946-bf56-7be5c68f9a94-kube-api-access-7lpgv\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-systemd-units\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-os-release\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-socket-dir-parent\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-multus-certs\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.471982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-etc-kubernetes\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.472765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471321 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-sys\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.472765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.472765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-log-socket\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.472765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-var-lib-kubelet\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.472765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfjl\" (UniqueName: \"kubernetes.io/projected/334b78c8-0fb9-46b2-942d-b634710b26b7-kube-api-access-drfjl\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.472765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.471598 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 10:27:21.473832 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.473071 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zkpq5\"" Aug 04 10:27:21.473832 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.473290 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 10:27:21.473832 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.473517 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 10:27:21.482454 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.482431 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 10:27:21.515608 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.515575 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2p29g" Aug 04 10:27:21.529373 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.528929 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2p29g" Aug 04 10:27:21.532251 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.532205 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" event={"ID":"de1bfd90085cee16926246b78cba2cc2","Type":"ContainerStarted","Data":"f59528aa6da1daf14972f960961970bb864430b7435040dc112ff7ca037c1866"} Aug 04 10:27:21.533737 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.533715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" event={"ID":"5230571842e399889e9775bf9720e82e","Type":"ContainerStarted","Data":"a23734ddd4dddd7d79ab5c359b860100e02521521e639a70ad23bc5febf857d9"} Aug 04 10:27:21.558583 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.557906 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-sys\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-log-socket\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-var-lib-kubelet\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drfjl\" (UniqueName: \"kubernetes.io/projected/334b78c8-0fb9-46b2-942d-b634710b26b7-kube-api-access-drfjl\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-hosts-file\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7dd768f-e27c-413b-8aeb-fd7c20330835-serviceca\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-socket-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-cni-netd\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-modprobe-d\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-slash\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-sys-fs\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-run-ovn-kubernetes\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-cni-bin\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.573235 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572492 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-run-netns\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-ovn\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-env-overrides\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27psm\" (UniqueName: \"kubernetes.io/projected/4bce4a74-6633-4b34-8d8d-86c76d30dcce-kube-api-access-27psm\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef06db54-e625-4946-bf56-7be5c68f9a94-tmp\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l48ft\" (UniqueName: \"kubernetes.io/projected/a7dd768f-e27c-413b-8aeb-fd7c20330835-kube-api-access-l48ft\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x25kx\" (UniqueName: \"kubernetes.io/projected/642e5a13-95f9-42bd-a40e-d74063aa8d4b-kube-api-access-x25kx\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5sdj\" (UniqueName: \"kubernetes.io/projected/0fa238f6-a750-4b82-8c0c-d31e0fcda574-kube-api-access-k5sdj\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-system-cni-dir\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-cni-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-os-release\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysconfig\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-tuned\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/763e93ec-a5c0-419a-8a46-05ee3fe71399-agent-certs\") pod \"konnectivity-agent-6h49x\" (UID: \"763e93ec-a5c0-419a-8a46-05ee3fe71399\") " pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.574131 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-device-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-etc-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-daemon-config\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-run-netns\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572937 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-cni-netd\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-var-lib-kubelet\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573041 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-run\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573098 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-ovn\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573149 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-sys-fs\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573189 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-run-ovn-kubernetes\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-modprobe-d\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.572922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-run\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573103 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-log-socket\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573265 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-device-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.574915 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.573273 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573290 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-etc-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-sys\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573290 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573312 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573330 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-system-cni-dir\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.573339 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:22.073320597 +0000 UTC m=+3.024728145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573337 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-slash\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573366 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-os-release\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-cni-bin\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573403 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-socket-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysconfig\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-cni-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-cni-bin\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/334b78c8-0fb9-46b2-942d-b634710b26b7-host-slash\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-cni-bin\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.575781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-conf-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/334b78c8-0fb9-46b2-942d-b634710b26b7-host-slash\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-registration-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-conf-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/642e5a13-95f9-42bd-a40e-d74063aa8d4b-registration-dir\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-daemon-config\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0416d84-cde5-4c8a-a141-40796d09738b-ovn-node-metrics-cert\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.573960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-ovnkube-script-lib\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-cnibin\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bce4a74-6633-4b34-8d8d-86c76d30dcce-cni-binary-copy\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574090 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-netns\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-netns\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-kubelet\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-cnibin\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-ovnkube-config\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574214 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-kubelet\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.576586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-env-overrides\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-hostroot\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-cni-binary-copy\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-cni-multus\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysctl-d\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-systemd\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-lib-modules\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12828ebf-4f28-42bb-9882-cc4f93bee964-dbus\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7dd768f-e27c-413b-8aeb-fd7c20330835-host\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8n2t\" (UniqueName: \"kubernetes.io/projected/0f593643-7d48-4bfd-b9c4-1ba164f23119-kube-api-access-v8n2t\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bce4a74-6633-4b34-8d8d-86c76d30dcce-cni-binary-copy\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-kubernetes\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574643 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-ovnkube-config\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574704 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-kubernetes\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574729 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysctl-d\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-systemd\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-lib-modules\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.577155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574755 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574787 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-var-lib-cni-multus\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-systemd\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-node-log\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-cnibin\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-kubelet\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-k8s-cni-cncf-io\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/334b78c8-0fb9-46b2-942d-b634710b26b7-iptables-alerter-script\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.574984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12828ebf-4f28-42bb-9882-cc4f93bee964-kubelet-config\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/763e93ec-a5c0-419a-8a46-05ee3fe71399-konnectivity-ca\") pod \"konnectivity-agent-6h49x\" (UID: \"763e93ec-a5c0-419a-8a46-05ee3fe71399\") " pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-var-lib-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-cnibin\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jv4\" (UniqueName: \"kubernetes.io/projected/c0416d84-cde5-4c8a-a141-40796d09738b-kube-api-access-72jv4\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575072 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-k8s-cni-cncf-io\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-system-cni-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575143 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-system-cni-dir\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575182 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-host-kubelet\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575215 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-node-log\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.577760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-cni-binary-copy\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575249 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-hostroot\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-run-systemd\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-var-lib-openvswitch\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysctl-conf\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-host\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpgv\" (UniqueName: \"kubernetes.io/projected/ef06db54-e625-4946-bf56-7be5c68f9a94-kube-api-access-7lpgv\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/334b78c8-0fb9-46b2-942d-b634710b26b7-iptables-alerter-script\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-tmp-dir\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575574 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmlx6\" (UniqueName: \"kubernetes.io/projected/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-kube-api-access-mmlx6\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575597 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-host\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-systemd-units\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-os-release\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-socket-dir-parent\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-multus-certs\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.578545 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-etc-kubernetes\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-sysctl-conf\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575946 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0416d84-cde5-4c8a-a141-40796d09738b-systemd-units\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575952 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0416d84-cde5-4c8a-a141-40796d09738b-ovnkube-script-lib\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.575995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f593643-7d48-4bfd-b9c4-1ba164f23119-os-release\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.576034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-multus-socket-dir-parent\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.576037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-host-run-multus-certs\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.576070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bce4a74-6633-4b34-8d8d-86c76d30dcce-etc-kubernetes\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.576404 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f593643-7d48-4bfd-b9c4-1ba164f23119-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.578621 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef06db54-e625-4946-bf56-7be5c68f9a94-tmp\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.579458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.579384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0416d84-cde5-4c8a-a141-40796d09738b-ovn-node-metrics-cert\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.580972 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.580878 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef06db54-e625-4946-bf56-7be5c68f9a94-etc-tuned\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.597211 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.597190 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:21.597301 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.597215 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:21.597301 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.597228 2577 projected.go:196] Error preparing data for projected volume kube-api-access-v6zn2 for pod openshift-network-diagnostics/network-check-target-5gw46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:21.597301 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.597284 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2 podName:631b0a8d-ae6e-433b-9927-0d0546cce4ee nodeName:}" failed. No retries permitted until 2026-08-04 10:27:22.097265783 +0000 UTC m=+3.048673310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v6zn2" (UniqueName: "kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2") pod "network-check-target-5gw46" (UID: "631b0a8d-ae6e-433b-9927-0d0546cce4ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:21.598734 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.598710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfjl\" (UniqueName: \"kubernetes.io/projected/334b78c8-0fb9-46b2-942d-b634710b26b7-kube-api-access-drfjl\") pod \"iptables-alerter-bt7sr\" (UID: \"334b78c8-0fb9-46b2-942d-b634710b26b7\") " pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.599277 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.598933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x25kx\" (UniqueName: \"kubernetes.io/projected/642e5a13-95f9-42bd-a40e-d74063aa8d4b-kube-api-access-x25kx\") pod \"aws-ebs-csi-driver-node-jvd5l\" (UID: \"642e5a13-95f9-42bd-a40e-d74063aa8d4b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.599277 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.599036 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27psm\" (UniqueName: \"kubernetes.io/projected/4bce4a74-6633-4b34-8d8d-86c76d30dcce-kube-api-access-27psm\") pod \"multus-8jftd\" (UID: \"4bce4a74-6633-4b34-8d8d-86c76d30dcce\") " pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.599277 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.599225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8n2t\" (UniqueName: \"kubernetes.io/projected/0f593643-7d48-4bfd-b9c4-1ba164f23119-kube-api-access-v8n2t\") pod \"multus-additional-cni-plugins-p82pr\" (UID: \"0f593643-7d48-4bfd-b9c4-1ba164f23119\") " pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.601864 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.601826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5sdj\" (UniqueName: \"kubernetes.io/projected/0fa238f6-a750-4b82-8c0c-d31e0fcda574-kube-api-access-k5sdj\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:21.602128 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.602107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jv4\" (UniqueName: \"kubernetes.io/projected/c0416d84-cde5-4c8a-a141-40796d09738b-kube-api-access-72jv4\") pod \"ovnkube-node-clkh9\" (UID: \"c0416d84-cde5-4c8a-a141-40796d09738b\") " pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.606561 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.606542 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpgv\" (UniqueName: \"kubernetes.io/projected/ef06db54-e625-4946-bf56-7be5c68f9a94-kube-api-access-7lpgv\") pod \"tuned-5bx2n\" (UID: \"ef06db54-e625-4946-bf56-7be5c68f9a94\") " pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.676608 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12828ebf-4f28-42bb-9882-cc4f93bee964-dbus\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.676608 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7dd768f-e27c-413b-8aeb-fd7c20330835-host\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.676859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.676859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12828ebf-4f28-42bb-9882-cc4f93bee964-kubelet-config\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.676859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/763e93ec-a5c0-419a-8a46-05ee3fe71399-konnectivity-ca\") pod \"konnectivity-agent-6h49x\" (UID: \"763e93ec-a5c0-419a-8a46-05ee3fe71399\") " pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.676859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-tmp-dir\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.676859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmlx6\" (UniqueName: \"kubernetes.io/projected/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-kube-api-access-mmlx6\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.676859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-hosts-file\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.676859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7dd768f-e27c-413b-8aeb-fd7c20330835-serviceca\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.677127 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.676863 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:21.677127 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l48ft\" (UniqueName: \"kubernetes.io/projected/a7dd768f-e27c-413b-8aeb-fd7c20330835-kube-api-access-l48ft\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.677127 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:21.676934 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret podName:12828ebf-4f28-42bb-9882-cc4f93bee964 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:22.176913077 +0000 UTC m=+3.128320605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret") pod "global-pull-secret-syncer-wdm6f" (UID: "12828ebf-4f28-42bb-9882-cc4f93bee964") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:21.677127 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.676970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/763e93ec-a5c0-419a-8a46-05ee3fe71399-agent-certs\") pod \"konnectivity-agent-6h49x\" (UID: \"763e93ec-a5c0-419a-8a46-05ee3fe71399\") " pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.677127 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.677110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7dd768f-e27c-413b-8aeb-fd7c20330835-host\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.677288 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.677182 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12828ebf-4f28-42bb-9882-cc4f93bee964-dbus\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.677333 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.677297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12828ebf-4f28-42bb-9882-cc4f93bee964-kubelet-config\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:21.677419 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.677399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-hosts-file\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.677760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.677717 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-tmp-dir\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.677822 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.677774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7dd768f-e27c-413b-8aeb-fd7c20330835-serviceca\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.677895 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.677855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/763e93ec-a5c0-419a-8a46-05ee3fe71399-konnectivity-ca\") pod \"konnectivity-agent-6h49x\" (UID: \"763e93ec-a5c0-419a-8a46-05ee3fe71399\") " pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.679888 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.679864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/763e93ec-a5c0-419a-8a46-05ee3fe71399-agent-certs\") pod \"konnectivity-agent-6h49x\" (UID: \"763e93ec-a5c0-419a-8a46-05ee3fe71399\") " pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.690036 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.689956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmlx6\" (UniqueName: \"kubernetes.io/projected/40a3ad3e-821e-4f8f-bf56-aeedbf62b949-kube-api-access-mmlx6\") pod \"node-resolver-q4mkf\" (UID: \"40a3ad3e-821e-4f8f-bf56-aeedbf62b949\") " pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:21.690219 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.690199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l48ft\" (UniqueName: \"kubernetes.io/projected/a7dd768f-e27c-413b-8aeb-fd7c20330835-kube-api-access-l48ft\") pod \"node-ca-bznw4\" (UID: \"a7dd768f-e27c-413b-8aeb-fd7c20330835\") " pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.759104 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.759080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8jftd" Aug 04 10:27:21.773264 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.773243 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" Aug 04 10:27:21.785381 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.785361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" Aug 04 10:27:21.792868 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.792848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:21.801448 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.801430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bt7sr" Aug 04 10:27:21.809913 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.809890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p82pr" Aug 04 10:27:21.818493 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.818476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:21.825993 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.825969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bznw4" Aug 04 10:27:21.832722 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:21.832707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4mkf" Aug 04 10:27:22.079669 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:22.079590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:22.079827 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.079720 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:22.079827 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.079798 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:23.079778137 +0000 UTC m=+4.031185669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:22.180015 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:22.179984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:22.180160 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:22.180134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:22.180216 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.180158 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:22.180216 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.180181 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:22.180216 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.180190 2577 projected.go:196] Error preparing data for projected volume kube-api-access-v6zn2 for pod openshift-network-diagnostics/network-check-target-5gw46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:22.180335 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.180224 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:22.180335 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.180246 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2 podName:631b0a8d-ae6e-433b-9927-0d0546cce4ee nodeName:}" failed. No retries permitted until 2026-08-04 10:27:23.180227783 +0000 UTC m=+4.131635326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6zn2" (UniqueName: "kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2") pod "network-check-target-5gw46" (UID: "631b0a8d-ae6e-433b-9927-0d0546cce4ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:22.180335 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:22.180271 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret podName:12828ebf-4f28-42bb-9882-cc4f93bee964 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:23.180255548 +0000 UTC m=+4.131663075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret") pod "global-pull-secret-syncer-wdm6f" (UID: "12828ebf-4f28-42bb-9882-cc4f93bee964") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:22.341521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:22.341453 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 10:27:22.533112 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:22.533076 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 10:22:21 +0000 UTC" deadline="2028-04-24 16:36:15.711749234 +0000 UTC" Aug 04 10:27:22.533112 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:22.533104 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15102h8m53.178649412s" Aug 04 10:27:22.675415 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:22.675390 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bce4a74_6633_4b34_8d8d_86c76d30dcce.slice/crio-8a2ba6481a64f12b302b8d0233b6509d22824a46c239d23158c84c6df7f4c334 WatchSource:0}: Error finding container 8a2ba6481a64f12b302b8d0233b6509d22824a46c239d23158c84c6df7f4c334: Status 404 returned error can't find the container with id 8a2ba6481a64f12b302b8d0233b6509d22824a46c239d23158c84c6df7f4c334 Aug 04 10:27:22.680964 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:22.680941 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dd768f_e27c_413b_8aeb_fd7c20330835.slice/crio-09014a14520cfc91b3ae3775b4b1ea14ee15056ad67ef409aa7c734e39c19228 WatchSource:0}: Error finding container 09014a14520cfc91b3ae3775b4b1ea14ee15056ad67ef409aa7c734e39c19228: Status 404 returned error can't find the container with id 09014a14520cfc91b3ae3775b4b1ea14ee15056ad67ef409aa7c734e39c19228 Aug 04 10:27:22.682243 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:22.682190 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334b78c8_0fb9_46b2_942d_b634710b26b7.slice/crio-a8173c08523cdc7b7b012a43c4ff021f0297268d4559d0be8a5869a00b58c5c1 WatchSource:0}: Error finding container a8173c08523cdc7b7b012a43c4ff021f0297268d4559d0be8a5869a00b58c5c1: Status 404 returned error can't find the container with id a8173c08523cdc7b7b012a43c4ff021f0297268d4559d0be8a5869a00b58c5c1 Aug 04 10:27:22.683298 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:22.683272 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0416d84_cde5_4c8a_a141_40796d09738b.slice/crio-81eff3db5d963eca01f6305eca70afc84f049a89654e778f4bb784f04912bfb3 WatchSource:0}: Error finding container 81eff3db5d963eca01f6305eca70afc84f049a89654e778f4bb784f04912bfb3: Status 404 returned error can't find the container with id 81eff3db5d963eca01f6305eca70afc84f049a89654e778f4bb784f04912bfb3 Aug 04 10:27:22.687424 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:22.687404 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef06db54_e625_4946_bf56_7be5c68f9a94.slice/crio-63ee77c026f23b6209327ae8b867dd95551f58830af1c35ffd132d95d05ab168 WatchSource:0}: Error finding container 63ee77c026f23b6209327ae8b867dd95551f58830af1c35ffd132d95d05ab168: Status 404 returned error can't find the container with id 63ee77c026f23b6209327ae8b867dd95551f58830af1c35ffd132d95d05ab168 Aug 04 10:27:22.689490 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:22.688919 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f593643_7d48_4bfd_b9c4_1ba164f23119.slice/crio-5a02f1e789beb8ec261a683247cbc9d90b39985057a4a30c05af6a1673b1671e WatchSource:0}: Error finding container 5a02f1e789beb8ec261a683247cbc9d90b39985057a4a30c05af6a1673b1671e: Status 404 returned error can't find the container with id 5a02f1e789beb8ec261a683247cbc9d90b39985057a4a30c05af6a1673b1671e Aug 04 10:27:23.087077 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.086889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:23.087077 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.087039 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:23.087262 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.087132 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:25.087116831 +0000 UTC m=+6.038524357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:23.188141 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.188068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:23.188141 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.188122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:23.188333 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.188235 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:23.188333 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.188281 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:23.188333 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.188300 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:23.188333 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.188303 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret podName:12828ebf-4f28-42bb-9882-cc4f93bee964 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:25.188283225 +0000 UTC m=+6.139690759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret") pod "global-pull-secret-syncer-wdm6f" (UID: "12828ebf-4f28-42bb-9882-cc4f93bee964") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:23.188333 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.188312 2577 projected.go:196] Error preparing data for projected volume kube-api-access-v6zn2 for pod openshift-network-diagnostics/network-check-target-5gw46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:23.188556 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.188356 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2 podName:631b0a8d-ae6e-433b-9927-0d0546cce4ee nodeName:}" failed. No retries permitted until 2026-08-04 10:27:25.188342815 +0000 UTC m=+6.139750355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6zn2" (UniqueName: "kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2") pod "network-check-target-5gw46" (UID: "631b0a8d-ae6e-433b-9927-0d0546cce4ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:23.527452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.526991 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:23.527452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.527001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:23.527452 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.527109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:23.527452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.527142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:23.527452 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.527240 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:23.527452 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:23.527313 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:23.535696 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.535664 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 10:22:21 +0000 UTC" deadline="2028-02-26 08:47:11.832993527 +0000 UTC" Aug 04 10:27:23.535696 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.535695 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13702h19m48.29730281s" Aug 04 10:27:23.557877 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.557695 2577 generic.go:356] "Generic (PLEG): container finished" podID="de1bfd90085cee16926246b78cba2cc2" containerID="2f9da0c3241b2975f80e62d7bd49f8a67ba60b722edd2554f48d1ecbe8085363" exitCode=0 Aug 04 10:27:23.557877 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.557773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" event={"ID":"de1bfd90085cee16926246b78cba2cc2","Type":"ContainerDied","Data":"2f9da0c3241b2975f80e62d7bd49f8a67ba60b722edd2554f48d1ecbe8085363"} Aug 04 10:27:23.563124 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.563056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerStarted","Data":"5a02f1e789beb8ec261a683247cbc9d90b39985057a4a30c05af6a1673b1671e"} Aug 04 10:27:23.576999 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.576942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6h49x" event={"ID":"763e93ec-a5c0-419a-8a46-05ee3fe71399","Type":"ContainerStarted","Data":"e9dd6e3503bfa861f2ba7761c3d190512ddfa1cebb26c397935b4e8a9c8088c2"} Aug 04 10:27:23.581068 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.581036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4mkf" event={"ID":"40a3ad3e-821e-4f8f-bf56-aeedbf62b949","Type":"ContainerStarted","Data":"8f5b526308bbf425a94a194631c53b2cdcd2df41937417777686396b95cef250"} Aug 04 10:27:23.584965 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.584915 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bt7sr" event={"ID":"334b78c8-0fb9-46b2-942d-b634710b26b7","Type":"ContainerStarted","Data":"a8173c08523cdc7b7b012a43c4ff021f0297268d4559d0be8a5869a00b58c5c1"} Aug 04 10:27:23.596312 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.596267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bznw4" event={"ID":"a7dd768f-e27c-413b-8aeb-fd7c20330835","Type":"ContainerStarted","Data":"09014a14520cfc91b3ae3775b4b1ea14ee15056ad67ef409aa7c734e39c19228"} Aug 04 10:27:23.601056 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.600940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8jftd" event={"ID":"4bce4a74-6633-4b34-8d8d-86c76d30dcce","Type":"ContainerStarted","Data":"8a2ba6481a64f12b302b8d0233b6509d22824a46c239d23158c84c6df7f4c334"} Aug 04 10:27:23.607378 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.607351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" event={"ID":"5230571842e399889e9775bf9720e82e","Type":"ContainerStarted","Data":"1a18ec1a903328f2f53d0438776cd09212f95188f815184a60cb6a17fe87044d"} Aug 04 10:27:23.626598 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.626572 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" event={"ID":"ef06db54-e625-4946-bf56-7be5c68f9a94","Type":"ContainerStarted","Data":"63ee77c026f23b6209327ae8b867dd95551f58830af1c35ffd132d95d05ab168"} Aug 04 10:27:23.630594 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.630560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"81eff3db5d963eca01f6305eca70afc84f049a89654e778f4bb784f04912bfb3"} Aug 04 10:27:23.638777 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:23.638752 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" event={"ID":"642e5a13-95f9-42bd-a40e-d74063aa8d4b","Type":"ContainerStarted","Data":"64a5fd6466a4fc1de0efc23ea50382c942434babb5cfed9d6a8ec832a1bca4db"} Aug 04 10:27:24.655546 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:24.655466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" event={"ID":"de1bfd90085cee16926246b78cba2cc2","Type":"ContainerStarted","Data":"eeaa11825cfc993279bb0efdc39c51dd414da8f240c245b33d70ff585f8072e5"} Aug 04 10:27:24.682173 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:24.681641 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-198.ec2.internal" podStartSLOduration=3.68162053 podStartE2EDuration="3.68162053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:23.634578725 +0000 UTC m=+4.585986275" watchObservedRunningTime="2026-08-04 10:27:24.68162053 +0000 UTC m=+5.633028078" Aug 04 10:27:25.102638 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:25.102607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:25.102818 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.102741 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:25.102818 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.102809 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:29.10278987 +0000 UTC m=+10.054197404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:25.203152 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:25.203118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:25.203345 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:25.203175 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:25.203345 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.203258 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:25.203345 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.203292 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:25.203345 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.203312 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:25.203345 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.203324 2577 projected.go:196] Error preparing data for projected volume kube-api-access-v6zn2 for pod openshift-network-diagnostics/network-check-target-5gw46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:25.203345 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.203328 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret podName:12828ebf-4f28-42bb-9882-cc4f93bee964 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:29.203309392 +0000 UTC m=+10.154716934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret") pod "global-pull-secret-syncer-wdm6f" (UID: "12828ebf-4f28-42bb-9882-cc4f93bee964") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:25.203651 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.203381 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2 podName:631b0a8d-ae6e-433b-9927-0d0546cce4ee nodeName:}" failed. No retries permitted until 2026-08-04 10:27:29.203365907 +0000 UTC m=+10.154773438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6zn2" (UniqueName: "kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2") pod "network-check-target-5gw46" (UID: "631b0a8d-ae6e-433b-9927-0d0546cce4ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:25.526973 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:25.526948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:25.527150 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:25.526950 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:25.527786 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:25.527546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:25.527786 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.527569 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:25.527786 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.527669 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:25.527786 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:25.527748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:27.526446 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:27.526412 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:27.526921 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:27.526750 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:27.538694 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:27.537979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:27.538694 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:27.538145 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:27.538694 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:27.538219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:27.538694 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:27.538315 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:29.135365 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:29.135320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:29.135821 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.135499 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:29.135821 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.135559 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:37.135541221 +0000 UTC m=+18.086948750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:29.236593 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:29.236561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:29.236756 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:29.236665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:29.236852 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.236757 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:29.236852 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.236783 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:29.236852 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.236796 2577 projected.go:196] Error preparing data for projected volume kube-api-access-v6zn2 for pod openshift-network-diagnostics/network-check-target-5gw46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:29.236961 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.236766 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:29.236961 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.236870 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2 podName:631b0a8d-ae6e-433b-9927-0d0546cce4ee nodeName:}" failed. No retries permitted until 2026-08-04 10:27:37.236850402 +0000 UTC m=+18.188257944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6zn2" (UniqueName: "kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2") pod "network-check-target-5gw46" (UID: "631b0a8d-ae6e-433b-9927-0d0546cce4ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:29.236961 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.236909 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret podName:12828ebf-4f28-42bb-9882-cc4f93bee964 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:37.236892954 +0000 UTC m=+18.188300485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret") pod "global-pull-secret-syncer-wdm6f" (UID: "12828ebf-4f28-42bb-9882-cc4f93bee964") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:29.527113 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:29.526702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:29.527113 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.526820 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:29.527331 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:29.527192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:29.527331 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.527282 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:29.527446 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:29.527362 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:29.527517 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:29.527495 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:31.527156 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:31.526858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:31.527156 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:31.526911 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:31.527156 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:31.526858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:31.527156 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:31.526997 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:31.527156 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:31.527081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:31.527156 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:31.527131 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:33.526533 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:33.526505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:33.526988 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:33.526507 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:33.526988 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:33.526619 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:33.526988 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:33.526505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:33.526988 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:33.526700 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:33.526988 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:33.526852 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:35.526331 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:35.526301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:35.526859 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:35.526415 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:35.526859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:35.526437 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:35.526859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:35.526483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:35.526859 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:35.526569 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:35.526859 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:35.526635 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:37.204173 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:37.204131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:37.204606 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.204284 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:37.204606 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.204349 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:53.20432895 +0000 UTC m=+34.155736478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:37.305290 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:37.305262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:37.305466 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:37.305300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:37.305466 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.305417 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:37.305585 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.305489 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret podName:12828ebf-4f28-42bb-9882-cc4f93bee964 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:53.305468428 +0000 UTC m=+34.256875966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret") pod "global-pull-secret-syncer-wdm6f" (UID: "12828ebf-4f28-42bb-9882-cc4f93bee964") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:37.305585 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.305425 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:37.305585 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.305552 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:37.305585 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.305568 2577 projected.go:196] Error preparing data for projected volume kube-api-access-v6zn2 for pod openshift-network-diagnostics/network-check-target-5gw46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:37.305767 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.305620 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2 podName:631b0a8d-ae6e-433b-9927-0d0546cce4ee nodeName:}" failed. No retries permitted until 2026-08-04 10:27:53.305608549 +0000 UTC m=+34.257016097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6zn2" (UniqueName: "kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2") pod "network-check-target-5gw46" (UID: "631b0a8d-ae6e-433b-9927-0d0546cce4ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:37.526028 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:37.525999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:37.526218 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:37.526139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:37.526218 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.526149 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:37.526344 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:37.526239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:37.526344 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.526307 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:37.526442 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:37.526361 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:39.528562 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:39.528535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:39.529033 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:39.528535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:39.529033 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:39.528654 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:39.529033 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:39.528522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:39.529033 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:39.528717 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:39.529033 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:39.528802 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:40.690387 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.690210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bznw4" event={"ID":"a7dd768f-e27c-413b-8aeb-fd7c20330835","Type":"ContainerStarted","Data":"4d7d744bffa831ec5f156fd05059631dff14df92c82cbce1df106e225abf8d3c"} Aug 04 10:27:40.691886 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.691788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8jftd" event={"ID":"4bce4a74-6633-4b34-8d8d-86c76d30dcce","Type":"ContainerStarted","Data":"ec7c228d2217e9f177945049382da4b197f50581cb2ec2a0cc662412c5f10ccf"} Aug 04 10:27:40.693562 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.693493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" event={"ID":"ef06db54-e625-4946-bf56-7be5c68f9a94","Type":"ContainerStarted","Data":"d4e0c9687ab42f5f63639d51e08d7e3c443641f6baabc8241f9c7e5d91364927"} Aug 04 10:27:40.696299 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.696269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"60f0efccab6091fa5107b962fcfe49c7012d69767bfc45960e1683856b1f0ab1"} Aug 04 10:27:40.700265 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.700225 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6h49x" event={"ID":"763e93ec-a5c0-419a-8a46-05ee3fe71399","Type":"ContainerStarted","Data":"bdbf9c3b038a8cc6dd08e3093bb74b81c3b289f147da714e6b8f3570cc4fc7bd"} Aug 04 10:27:40.706931 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.706634 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bznw4" podStartSLOduration=3.883611272 podStartE2EDuration="21.706617092s" totalImagesPullingTime="17.82300582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:40.706216615 +0000 UTC m=+21.657624165" watchObservedRunningTime="2026-08-04 10:27:40.706617092 +0000 UTC m=+21.658024644" Aug 04 10:27:40.706931 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.706727 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-198.ec2.internal" podStartSLOduration=19.706718445 podStartE2EDuration="19.706718445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:24.682139501 +0000 UTC m=+5.633547058" watchObservedRunningTime="2026-08-04 10:27:40.706718445 +0000 UTC m=+21.658125997" Aug 04 10:27:40.747972 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.746855 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8jftd" podStartSLOduration=3.8543295520000003 podStartE2EDuration="21.746817218s" totalImagesPullingTime="17.892487666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:40.729655315 +0000 UTC m=+21.681062868" watchObservedRunningTime="2026-08-04 10:27:40.746817218 +0000 UTC m=+21.698224768" Aug 04 10:27:40.747972 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.747634 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5bx2n" podStartSLOduration=3.930331806 podStartE2EDuration="21.747618376s" totalImagesPullingTime="17.81728657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:40.745643267 +0000 UTC m=+21.697050817" watchObservedRunningTime="2026-08-04 10:27:40.747618376 +0000 UTC m=+21.699025927" Aug 04 10:27:40.760859 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.760797 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6h49x" podStartSLOduration=7.7310517579999996 podStartE2EDuration="21.760779629s" totalImagesPullingTime="14.029727871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:40.760577 +0000 UTC m=+21.711984552" watchObservedRunningTime="2026-08-04 10:27:40.760779629 +0000 UTC m=+21.712187178" Aug 04 10:27:40.832933 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.832735 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:40.833300 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:40.833282 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:41.526138 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.526113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:41.526300 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.526113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:41.526300 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:41.526242 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:41.526300 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.526266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:41.526445 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:41.526349 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:41.526445 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:41.526406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:41.703771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.703741 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f593643-7d48-4bfd-b9c4-1ba164f23119" containerID="0c3a9ef68fbc89249238a91b74f9075254f22e7ab7cec8211ed103c8e2a18351" exitCode=0 Aug 04 10:27:41.704370 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.703842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerDied","Data":"0c3a9ef68fbc89249238a91b74f9075254f22e7ab7cec8211ed103c8e2a18351"} Aug 04 10:27:41.705278 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.705253 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4mkf" event={"ID":"40a3ad3e-821e-4f8f-bf56-aeedbf62b949","Type":"ContainerStarted","Data":"babd2caa2319362e6cf5dbf481af51748150fa6a7e9e45fad6f9a947453f9a51"} Aug 04 10:27:41.706899 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.706873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bt7sr" event={"ID":"334b78c8-0fb9-46b2-942d-b634710b26b7","Type":"ContainerStarted","Data":"0ee1ce58173a13852f8bd52a1efa4d3f040462ad6e5aca5a405be4902b97b424"} Aug 04 10:27:41.710883 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.710863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"2aeef27bdb41f4643a6658f36745e7808f4f4504534372d8ff525a0512611834"} Aug 04 10:27:41.710986 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.710927 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"6a9e1484f55227bc9e5841e17a813a1481c7d33f3262ee1752fc70e529f14b39"} Aug 04 10:27:41.710986 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.710974 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"dfd5fc7d13901cbf2c0be66f814ee767988602cafb2ceb6b2aa72dd4796dc4f3"} Aug 04 10:27:41.711065 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.711019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"60af769b6e0c2d6fff66f01fcd4b87d6f594ba8f956a5ffa247632145f739ac7"} Aug 04 10:27:41.711065 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.711055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"5312e61086f4ec6cceab973fe9b836aeec1c927d708afc1ab9036c9f4b1d53ac"} Aug 04 10:27:41.712270 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.712250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" event={"ID":"642e5a13-95f9-42bd-a40e-d74063aa8d4b","Type":"ContainerStarted","Data":"a7817e480257951bce3cfb857bd3739d999d1cc7a2f56e0b310311a295c30c2f"} Aug 04 10:27:41.712779 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.712753 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:41.713126 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.713111 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6h49x" Aug 04 10:27:41.740723 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.740685 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bt7sr" podStartSLOduration=4.919293842 podStartE2EDuration="22.740670726s" totalImagesPullingTime="17.821376884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:41.740386385 +0000 UTC m=+22.691793933" watchObservedRunningTime="2026-08-04 10:27:41.740670726 +0000 UTC m=+22.692078305" Aug 04 10:27:41.776847 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:41.776767 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q4mkf" podStartSLOduration=3.95706564 podStartE2EDuration="21.776751671s" totalImagesPullingTime="17.819686031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:41.776744673 +0000 UTC m=+22.728152225" watchObservedRunningTime="2026-08-04 10:27:41.776751671 +0000 UTC m=+22.728159221" Aug 04 10:27:42.341262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:42.341230 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 10:27:42.525547 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:42.525425 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T10:27:42.341261134Z","UUID":"4d2c8516-c820-4221-b08a-8d4dcc8c517b","Handler":null,"Name":"","Endpoint":""} Aug 04 10:27:42.526646 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:42.526630 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 10:27:42.526724 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:42.526661 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 10:27:42.715473 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:42.715440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" event={"ID":"642e5a13-95f9-42bd-a40e-d74063aa8d4b","Type":"ContainerStarted","Data":"a43d51e1533ff0458afb0f6acfc78355f074d405fe3f458b01556eef138a4003"} Aug 04 10:27:43.526147 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:43.526118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:43.526345 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:43.526129 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:43.526345 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:43.526226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:43.526345 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:43.526239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:43.526501 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:43.526341 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:43.526501 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:43.526410 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:43.721002 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:43.720933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"b8eb7008e80e694bf4cb94c8eb706f97d76f0d05e6da897c84f797e538ec677f"} Aug 04 10:27:44.725188 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:44.724998 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" event={"ID":"642e5a13-95f9-42bd-a40e-d74063aa8d4b","Type":"ContainerStarted","Data":"ad6a3b3f8947d4980fd15fb9b822367c5914c9c5f55d047bc05b1dc220449958"} Aug 04 10:27:44.743253 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:44.743212 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvd5l" podStartSLOduration=4.963726052 podStartE2EDuration="25.743198342s" totalImagesPullingTime="20.77947229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:44.743063516 +0000 UTC m=+25.694471064" watchObservedRunningTime="2026-08-04 10:27:44.743198342 +0000 UTC m=+25.694605890" Aug 04 10:27:45.526476 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:45.526438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:45.526640 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:45.526438 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:45.526640 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:45.526564 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:45.526640 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:45.526584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:45.526800 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:45.526687 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:45.526800 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:45.526785 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:47.526484 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.526451 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:47.527201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.526567 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:47.527201 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.526580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:47.527201 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:47.526574 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:47.527201 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:47.526657 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:47.527201 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:47.526712 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:47.732840 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.732812 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f593643-7d48-4bfd-b9c4-1ba164f23119" containerID="031ce95b6f5a167615a06bcc34a498c73916fe897401633960fafb3713a88fef" exitCode=0 Aug 04 10:27:47.733021 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.732898 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerDied","Data":"031ce95b6f5a167615a06bcc34a498c73916fe897401633960fafb3713a88fef"} Aug 04 10:27:47.736568 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.736548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" event={"ID":"c0416d84-cde5-4c8a-a141-40796d09738b","Type":"ContainerStarted","Data":"1d49384fbe3d3a454b1d19156e0c24dcfb7057beb96a0089ba5d53e5820e3532"} Aug 04 10:27:47.736808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.736785 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:47.751118 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.751102 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:47.790068 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:47.789998 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" podStartSLOduration=10.917852757 podStartE2EDuration="28.789986203s" totalImagesPullingTime="17.872133446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:47.789582856 +0000 UTC m=+28.740990417" watchObservedRunningTime="2026-08-04 10:27:47.789986203 +0000 UTC m=+28.741393751" Aug 04 10:27:48.724709 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.724674 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdm6f"] Aug 04 10:27:48.725161 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.724820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:48.725161 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:48.724940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:48.727404 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.727378 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gw46"] Aug 04 10:27:48.727508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.727454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:48.727555 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:48.727514 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:48.730028 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.730004 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xrhhd"] Aug 04 10:27:48.730116 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.730106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:48.730218 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:48.730201 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:48.739236 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.739208 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:48.739351 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.739289 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:48.755792 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:48.755768 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:27:49.743068 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:49.743037 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f593643-7d48-4bfd-b9c4-1ba164f23119" containerID="8b5f3848fcd54e28e33dfc521e768359a63877099d171ec0412aca3e67ab6128" exitCode=0 Aug 04 10:27:49.743432 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:49.743122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerDied","Data":"8b5f3848fcd54e28e33dfc521e768359a63877099d171ec0412aca3e67ab6128"} Aug 04 10:27:50.526505 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:50.526302 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:50.526680 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:50.526311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:50.526680 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:50.526600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:50.526680 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:50.526311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:50.526863 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:50.526672 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:50.526863 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:50.526766 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:51.749227 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:51.749194 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f593643-7d48-4bfd-b9c4-1ba164f23119" containerID="2f5b389b716d611e8a3294aceaa2289f2a1c9b8b21da24cb36c3eb449900735f" exitCode=0 Aug 04 10:27:51.749642 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:51.749262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerDied","Data":"2f5b389b716d611e8a3294aceaa2289f2a1c9b8b21da24cb36c3eb449900735f"} Aug 04 10:27:52.526011 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:52.525970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:52.526168 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:52.525971 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:52.526168 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:52.526077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdm6f" podUID="12828ebf-4f28-42bb-9882-cc4f93bee964" Aug 04 10:27:52.526168 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:52.526151 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gw46" podUID="631b0a8d-ae6e-433b-9927-0d0546cce4ee" Aug 04 10:27:52.526265 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:52.525970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:52.526265 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:52.526227 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xrhhd" podUID="0fa238f6-a750-4b82-8c0c-d31e0fcda574" Aug 04 10:27:53.224317 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.224274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:53.224766 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.224422 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:53.224766 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.224481 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.224464732 +0000 UTC m=+66.175872260 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 10:27:53.325110 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.325049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:53.325110 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.325100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:53.325280 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.325179 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:53.325280 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.325233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret podName:12828ebf-4f28-42bb-9882-cc4f93bee964 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.325218418 +0000 UTC m=+66.276625946 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret") pod "global-pull-secret-syncer-wdm6f" (UID: "12828ebf-4f28-42bb-9882-cc4f93bee964") : object "kube-system"/"original-pull-secret" not registered Aug 04 10:27:53.325280 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.325245 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 10:27:53.325280 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.325263 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 10:27:53.325280 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.325274 2577 projected.go:196] Error preparing data for projected volume kube-api-access-v6zn2 for pod openshift-network-diagnostics/network-check-target-5gw46: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:53.325469 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.325319 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2 podName:631b0a8d-ae6e-433b-9927-0d0546cce4ee nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.325305837 +0000 UTC m=+66.276713372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6zn2" (UniqueName: "kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2") pod "network-check-target-5gw46" (UID: "631b0a8d-ae6e-433b-9927-0d0546cce4ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 10:27:53.346950 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.346929 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-198.ec2.internal" event="NodeReady" Aug 04 10:27:53.347067 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.347041 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 10:27:53.395172 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.395149 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5979db58b7-mr9p5"] Aug 04 10:27:53.399363 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.399345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.401583 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.401562 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wt4lh\"" Aug 04 10:27:53.401686 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.401584 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 10:27:53.401813 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.401798 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 10:27:53.401913 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.401801 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 10:27:53.406751 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.406321 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h8frs"] Aug 04 10:27:53.408394 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.408206 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 10:27:53.410340 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.410324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.412573 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.412554 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wlm2z\"" Aug 04 10:27:53.412573 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.412561 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 10:27:53.412817 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.412632 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 10:27:53.414917 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.414579 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rfcm6"] Aug 04 10:27:53.418598 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.418579 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5979db58b7-mr9p5"] Aug 04 10:27:53.418718 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.418705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.421250 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.421230 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 10:27:53.421533 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.421515 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 10:27:53.422194 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.422011 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8cx65\"" Aug 04 10:27:53.422194 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.422024 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 10:27:53.425929 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.425759 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h8frs"] Aug 04 10:27:53.428650 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.428604 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rfcm6"] Aug 04 10:27:53.527183 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.527306 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c508ad-842a-41f1-9fc0-79df29976a48-tmp-dir\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.527306 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-registry-certificates\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527306 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16a59ee9-ea69-403e-8ee7-06492cb61c78-cert\") pod \"ingress-canary-rfcm6\" (UID: \"16a59ee9-ea69-403e-8ee7-06492cb61c78\") " pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.527306 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-image-registry-private-configuration\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527477 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-trusted-ca\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527477 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-bound-sa-token\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527477 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgln\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-kube-api-access-4wgln\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527477 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39c508ad-842a-41f1-9fc0-79df29976a48-config-volume\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.527477 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527477 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/777e50d7-624b-4a82-99c5-56675f43183a-ca-trust-extracted\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527707 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-installation-pull-secrets\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.527707 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtlw\" (UniqueName: \"kubernetes.io/projected/39c508ad-842a-41f1-9fc0-79df29976a48-kube-api-access-tmtlw\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.527707 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.527564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw8h8\" (UniqueName: \"kubernetes.io/projected/16a59ee9-ea69-403e-8ee7-06492cb61c78-kube-api-access-qw8h8\") pod \"ingress-canary-rfcm6\" (UID: \"16a59ee9-ea69-403e-8ee7-06492cb61c78\") " pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.628861 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.628795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39c508ad-842a-41f1-9fc0-79df29976a48-config-volume\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.628861 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.628826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.628862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/777e50d7-624b-4a82-99c5-56675f43183a-ca-trust-extracted\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.628905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-installation-pull-secrets\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.628930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtlw\" (UniqueName: \"kubernetes.io/projected/39c508ad-842a-41f1-9fc0-79df29976a48-kube-api-access-tmtlw\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.628955 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.628961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw8h8\" (UniqueName: \"kubernetes.io/projected/16a59ee9-ea69-403e-8ee7-06492cb61c78-kube-api-access-qw8h8\") pod \"ingress-canary-rfcm6\" (UID: \"16a59ee9-ea69-403e-8ee7-06492cb61c78\") " pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.628971 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5979db58b7-mr9p5: secret "image-registry-tls" not found Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.629029 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.629026 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls podName:777e50d7-624b-4a82-99c5-56675f43183a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:54.129008524 +0000 UTC m=+35.080416062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls") pod "image-registry-5979db58b7-mr9p5" (UID: "777e50d7-624b-4a82-99c5-56675f43183a") : secret "image-registry-tls" not found Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c508ad-842a-41f1-9fc0-79df29976a48-tmp-dir\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.629067 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-registry-certificates\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:53.629101 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls podName:39c508ad-842a-41f1-9fc0-79df29976a48 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:54.129089546 +0000 UTC m=+35.080497095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls") pod "dns-default-h8frs" (UID: "39c508ad-842a-41f1-9fc0-79df29976a48") : secret "dns-default-metrics-tls" not found Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16a59ee9-ea69-403e-8ee7-06492cb61c78-cert\") pod \"ingress-canary-rfcm6\" (UID: \"16a59ee9-ea69-403e-8ee7-06492cb61c78\") " pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-image-registry-private-configuration\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-trusted-ca\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-bound-sa-token\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgln\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-kube-api-access-4wgln\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.629361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/777e50d7-624b-4a82-99c5-56675f43183a-ca-trust-extracted\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.630117 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/39c508ad-842a-41f1-9fc0-79df29976a48-config-volume\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.630117 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-registry-certificates\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.630117 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.629669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c508ad-842a-41f1-9fc0-79df29976a48-tmp-dir\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.631055 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.631010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-trusted-ca\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.633574 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.633532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-installation-pull-secrets\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.633574 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.633564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-image-registry-private-configuration\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.633766 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.633719 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16a59ee9-ea69-403e-8ee7-06492cb61c78-cert\") pod \"ingress-canary-rfcm6\" (UID: \"16a59ee9-ea69-403e-8ee7-06492cb61c78\") " pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.634635 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.634575 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:27:53.640140 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.639705 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw8h8\" (UniqueName: \"kubernetes.io/projected/16a59ee9-ea69-403e-8ee7-06492cb61c78-kube-api-access-qw8h8\") pod \"ingress-canary-rfcm6\" (UID: \"16a59ee9-ea69-403e-8ee7-06492cb61c78\") " pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.640140 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.639820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-bound-sa-token\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.640140 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.640123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgln\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-kube-api-access-4wgln\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:53.640420 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.640399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtlw\" (UniqueName: \"kubernetes.io/projected/39c508ad-842a-41f1-9fc0-79df29976a48-kube-api-access-tmtlw\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:53.729385 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.729361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rfcm6" Aug 04 10:27:53.741849 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.741810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:27:53.879555 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:53.879311 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rfcm6"] Aug 04 10:27:53.884964 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:27:53.884933 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a59ee9_ea69_403e_8ee7_06492cb61c78.slice/crio-532e91e0c5761f7f4790907a7e880c103f6f7dd1f6d6ccfa9e64805ff59f0f65 WatchSource:0}: Error finding container 532e91e0c5761f7f4790907a7e880c103f6f7dd1f6d6ccfa9e64805ff59f0f65: Status 404 returned error can't find the container with id 532e91e0c5761f7f4790907a7e880c103f6f7dd1f6d6ccfa9e64805ff59f0f65 Aug 04 10:27:54.132994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.132910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:54.132994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.132982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:54.133203 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:54.133062 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:27:54.133203 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:54.133104 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:27:54.133203 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:54.133117 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5979db58b7-mr9p5: secret "image-registry-tls" not found Aug 04 10:27:54.133203 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:54.133125 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls podName:39c508ad-842a-41f1-9fc0-79df29976a48 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:55.133111464 +0000 UTC m=+36.084518991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls") pod "dns-default-h8frs" (UID: "39c508ad-842a-41f1-9fc0-79df29976a48") : secret "dns-default-metrics-tls" not found Aug 04 10:27:54.133203 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:54.133162 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls podName:777e50d7-624b-4a82-99c5-56675f43183a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:55.1331477 +0000 UTC m=+36.084555246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls") pod "image-registry-5979db58b7-mr9p5" (UID: "777e50d7-624b-4a82-99c5-56675f43183a") : secret "image-registry-tls" not found Aug 04 10:27:54.526069 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.526043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:27:54.526776 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.526043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:27:54.526776 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.526177 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:27:54.528687 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.528664 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p48mn\"" Aug 04 10:27:54.529350 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.529120 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 10:27:54.529350 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.529130 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 10:27:54.529350 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.529153 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 10:27:54.529350 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.529228 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clvkl\"" Aug 04 10:27:54.529758 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.529742 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 10:27:54.758525 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:54.758487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rfcm6" event={"ID":"16a59ee9-ea69-403e-8ee7-06492cb61c78","Type":"ContainerStarted","Data":"532e91e0c5761f7f4790907a7e880c103f6f7dd1f6d6ccfa9e64805ff59f0f65"} Aug 04 10:27:55.140383 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:55.140353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:55.140571 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:55.140422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:55.140571 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:55.140521 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:27:55.140571 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:55.140529 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:27:55.140571 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:55.140550 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5979db58b7-mr9p5: secret "image-registry-tls" not found Aug 04 10:27:55.140711 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:55.140576 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls podName:39c508ad-842a-41f1-9fc0-79df29976a48 nodeName:}" failed. No retries permitted until 2026-08-04 10:27:57.140562065 +0000 UTC m=+38.091969592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls") pod "dns-default-h8frs" (UID: "39c508ad-842a-41f1-9fc0-79df29976a48") : secret "dns-default-metrics-tls" not found Aug 04 10:27:55.140711 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:55.140598 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls podName:777e50d7-624b-4a82-99c5-56675f43183a nodeName:}" failed. No retries permitted until 2026-08-04 10:27:57.14058209 +0000 UTC m=+38.091989623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls") pod "image-registry-5979db58b7-mr9p5" (UID: "777e50d7-624b-4a82-99c5-56675f43183a") : secret "image-registry-tls" not found Aug 04 10:27:57.155892 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:57.155856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:27:57.156358 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:57.155941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:27:57.156358 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:57.156013 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:27:57.156358 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:57.156030 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5979db58b7-mr9p5: secret "image-registry-tls" not found Aug 04 10:27:57.156358 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:57.156069 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:27:57.156358 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:57.156092 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls podName:777e50d7-624b-4a82-99c5-56675f43183a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:01.156071445 +0000 UTC m=+42.107478982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls") pod "image-registry-5979db58b7-mr9p5" (UID: "777e50d7-624b-4a82-99c5-56675f43183a") : secret "image-registry-tls" not found Aug 04 10:27:57.156358 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:27:57.156131 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls podName:39c508ad-842a-41f1-9fc0-79df29976a48 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:01.156113737 +0000 UTC m=+42.107521280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls") pod "dns-default-h8frs" (UID: "39c508ad-842a-41f1-9fc0-79df29976a48") : secret "dns-default-metrics-tls" not found Aug 04 10:27:57.911405 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:57.911328 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:27:58.012506 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:58.012478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:27:58.770238 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:58.770210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rfcm6" event={"ID":"16a59ee9-ea69-403e-8ee7-06492cb61c78","Type":"ContainerStarted","Data":"58a208e995669bb0a5e72847042bff33d5930ac107e1fcc5f3acda677d8530ec"} Aug 04 10:27:58.772590 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:58.772566 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f593643-7d48-4bfd-b9c4-1ba164f23119" containerID="c61a6358048388afc0fc486850b1d24498b5a6efdd640eb830ffd26ee789ee4c" exitCode=0 Aug 04 10:27:58.772708 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:58.772601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerDied","Data":"c61a6358048388afc0fc486850b1d24498b5a6efdd640eb830ffd26ee789ee4c"} Aug 04 10:27:58.802123 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:58.802077 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rfcm6" podStartSLOduration=1.916312303 podStartE2EDuration="5.802064613s" totalImagesPullingTime="3.88575231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:27:58.801379425 +0000 UTC m=+39.752786974" watchObservedRunningTime="2026-08-04 10:27:58.802064613 +0000 UTC m=+39.753472161" Aug 04 10:27:59.776975 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:59.776938 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f593643-7d48-4bfd-b9c4-1ba164f23119" containerID="aca36a5136feb83b084109e334e369bb1c1e02d4f66a578d3907923dac9b1a6f" exitCode=0 Aug 04 10:27:59.777328 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:27:59.777015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerDied","Data":"aca36a5136feb83b084109e334e369bb1c1e02d4f66a578d3907923dac9b1a6f"} Aug 04 10:28:00.781697 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:00.781663 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p82pr" event={"ID":"0f593643-7d48-4bfd-b9c4-1ba164f23119","Type":"ContainerStarted","Data":"676aa4a9e4210c3eec1f68bb15cebb349116e93795e9754cfd089929146b908a"} Aug 04 10:28:00.936988 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:00.936961 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:28:01.038728 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:01.038662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:28:01.188184 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:01.188157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:28:01.188340 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:01.188202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:28:01.188340 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:01.188289 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:01.188401 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:01.188341 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls podName:39c508ad-842a-41f1-9fc0-79df29976a48 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:09.188326578 +0000 UTC m=+50.139734105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls") pod "dns-default-h8frs" (UID: "39c508ad-842a-41f1-9fc0-79df29976a48") : secret "dns-default-metrics-tls" not found Aug 04 10:28:01.188401 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:01.188291 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:01.188401 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:01.188382 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5979db58b7-mr9p5: secret "image-registry-tls" not found Aug 04 10:28:01.188493 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:01.188427 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls podName:777e50d7-624b-4a82-99c5-56675f43183a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:09.188416194 +0000 UTC m=+50.139823722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls") pod "image-registry-5979db58b7-mr9p5" (UID: "777e50d7-624b-4a82-99c5-56675f43183a") : secret "image-registry-tls" not found Aug 04 10:28:03.430223 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:03.430193 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:28:03.501523 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:03.501486 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:28:04.375105 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:04.375071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:28:04.430778 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:04.430748 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" limit=1024 Aug 04 10:28:09.240671 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:09.240637 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:28:09.241061 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:09.240682 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:09.241061 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:09.240700 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5979db58b7-mr9p5: secret "image-registry-tls" not found Aug 04 10:28:09.241061 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:09.240713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:28:09.241061 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:09.240760 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls podName:777e50d7-624b-4a82-99c5-56675f43183a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.240745888 +0000 UTC m=+66.192153415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls") pod "image-registry-5979db58b7-mr9p5" (UID: "777e50d7-624b-4a82-99c5-56675f43183a") : secret "image-registry-tls" not found Aug 04 10:28:09.241061 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:09.240798 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:09.241061 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:09.240882 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls podName:39c508ad-842a-41f1-9fc0-79df29976a48 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.240870911 +0000 UTC m=+66.192278443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls") pod "dns-default-h8frs" (UID: "39c508ad-842a-41f1-9fc0-79df29976a48") : secret "dns-default-metrics-tls" not found Aug 04 10:28:15.154247 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.154188 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p82pr" podStartSLOduration=24.565603151 podStartE2EDuration="56.154172191s" totalImagesPullingTime="31.58856904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:00.811067571 +0000 UTC m=+41.762475131" watchObservedRunningTime="2026-08-04 10:28:15.154172191 +0000 UTC m=+56.105579739" Aug 04 10:28:15.154663 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.154403 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-587b49775d-8b8rl"] Aug 04 10:28:15.157369 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.157351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.160481 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.160453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 10:28:15.161157 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.160959 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 10:28:15.161542 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.161520 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.161542 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.161534 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 10:28:15.161687 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.161564 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.161687 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.161624 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq"] Aug 04 10:28:15.161797 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.161780 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wxfrj\"" Aug 04 10:28:15.161857 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.161782 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 10:28:15.164148 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.164130 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-5kbmf"] Aug 04 10:28:15.164231 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.164216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" Aug 04 10:28:15.166712 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.166691 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm"] Aug 04 10:28:15.166787 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.166779 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.166848 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.166808 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.166902 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.166893 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.167098 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.167080 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kmvdx\"" Aug 04 10:28:15.169516 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.169489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:15.172816 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.172801 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.173282 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.173267 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.173540 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.173514 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 04 10:28:15.173605 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.173548 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 10:28:15.173605 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.173558 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nbfr5\"" Aug 04 10:28:15.173605 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.173587 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.173750 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.173695 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-64mrf\"" Aug 04 10:28:15.173878 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.173864 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.178780 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.178766 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 10:28:15.181435 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.181416 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 10:28:15.183523 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-default-certificate\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.183620 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183540 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-5kbmf"] Aug 04 10:28:15.183620 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-stats-auth\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.183620 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.183776 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rltdt\" (UniqueName: \"kubernetes.io/projected/b16fa908-9c26-438c-bf68-9e44b999a1fb-kube-api-access-rltdt\") pod \"volume-data-source-validator-5b8776669-2wdxq\" (UID: \"b16fa908-9c26-438c-bf68-9e44b999a1fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" Aug 04 10:28:15.183776 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8cn\" (UniqueName: \"kubernetes.io/projected/bd849d55-f17a-4a9c-bc88-5e60089d83ed-kube-api-access-nq8cn\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:15.183776 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rhn\" (UniqueName: \"kubernetes.io/projected/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-kube-api-access-n6rhn\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.183776 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-trusted-ca\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.183976 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.183976 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-config\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.183976 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-serving-cert\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.183976 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.183951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:15.184221 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.184018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9wf\" (UniqueName: \"kubernetes.io/projected/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-kube-api-access-tj9wf\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.192799 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.192779 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm"] Aug 04 10:28:15.226407 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.226380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-587b49775d-8b8rl"] Aug 04 10:28:15.248110 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.248089 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq"] Aug 04 10:28:15.260687 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.260671 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml"] Aug 04 10:28:15.263607 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.263592 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn"] Aug 04 10:28:15.263711 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.263696 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" Aug 04 10:28:15.266139 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.266120 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j5g6n\"" Aug 04 10:28:15.267560 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.267543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.269167 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.269148 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.269256 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.269246 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.269597 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.269577 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-l7kqp\"" Aug 04 10:28:15.269647 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.269602 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 10:28:15.269768 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.269740 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 10:28:15.284697 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284679 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml"] Aug 04 10:28:15.284811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-default-certificate\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.284811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.284811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzs4\" (UniqueName: \"kubernetes.io/projected/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-kube-api-access-phzs4\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.284994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.284994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8cn\" (UniqueName: \"kubernetes.io/projected/bd849d55-f17a-4a9c-bc88-5e60089d83ed-kube-api-access-nq8cn\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:15.284994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-trusted-ca\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.284994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.284991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-config\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.285174 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-serving-cert\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.285174 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:15.285174 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.285174 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9wf\" (UniqueName: \"kubernetes.io/projected/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-kube-api-access-tj9wf\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.285174 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-stats-auth\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.285174 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rltdt\" (UniqueName: \"kubernetes.io/projected/b16fa908-9c26-438c-bf68-9e44b999a1fb-kube-api-access-rltdt\") pod \"volume-data-source-validator-5b8776669-2wdxq\" (UID: \"b16fa908-9c26-438c-bf68-9e44b999a1fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" Aug 04 10:28:15.285456 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.285101 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:28:15.285456 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.285390 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:15.785371789 +0000 UTC m=+56.736779322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : secret "router-metrics-certs-default" not found Aug 04 10:28:15.285559 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.285460 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:28:15.285559 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.285525 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls podName:bd849d55-f17a-4a9c-bc88-5e60089d83ed nodeName:}" failed. No retries permitted until 2026-08-04 10:28:15.785503606 +0000 UTC m=+56.736911145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-4g2wm" (UID: "bd849d55-f17a-4a9c-bc88-5e60089d83ed") : secret "samples-operator-tls" not found Aug 04 10:28:15.285662 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.285712 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.285666 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:15.785653441 +0000 UTC m=+56.737060985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : configmap references non-existent config key: service-ca.crt Aug 04 10:28:15.285857 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6lf\" (UniqueName: \"kubernetes.io/projected/8bcc4a9d-4c90-472f-9821-8ade3abf9587-kube-api-access-5w6lf\") pod \"network-check-source-65bdbbff45-4cgml\" (UID: \"8bcc4a9d-4c90-472f-9821-8ade3abf9587\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" Aug 04 10:28:15.285994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.285974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rhn\" (UniqueName: \"kubernetes.io/projected/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-kube-api-access-n6rhn\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.286164 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.286140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-trusted-ca\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.286237 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.286210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-config\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.286877 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.286857 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn"] Aug 04 10:28:15.288870 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.288852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-serving-cert\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.289151 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.289136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-default-certificate\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.289214 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.289164 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-stats-auth\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.292811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.292792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8cn\" (UniqueName: \"kubernetes.io/projected/bd849d55-f17a-4a9c-bc88-5e60089d83ed-kube-api-access-nq8cn\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:15.294141 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.294124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9wf\" (UniqueName: \"kubernetes.io/projected/b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71-kube-api-access-tj9wf\") pod \"console-operator-5ccd6b445f-5kbmf\" (UID: \"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71\") " pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.299182 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.299160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rltdt\" (UniqueName: \"kubernetes.io/projected/b16fa908-9c26-438c-bf68-9e44b999a1fb-kube-api-access-rltdt\") pod \"volume-data-source-validator-5b8776669-2wdxq\" (UID: \"b16fa908-9c26-438c-bf68-9e44b999a1fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" Aug 04 10:28:15.302731 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.302711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rhn\" (UniqueName: \"kubernetes.io/projected/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-kube-api-access-n6rhn\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.349510 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.349494 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx"] Aug 04 10:28:15.353427 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.353414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.355508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.355478 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 10:28:15.355508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.355478 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-krbwq\"" Aug 04 10:28:15.355689 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.355526 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 10:28:15.355913 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.355895 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.356171 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.356158 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p"] Aug 04 10:28:15.356817 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.356802 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.358741 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.358726 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x"] Aug 04 10:28:15.358912 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.358893 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.360451 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.360436 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.361054 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.361037 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 10:28:15.361137 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.361122 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.361395 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.361380 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 10:28:15.361471 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.361437 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-ws6m5"] Aug 04 10:28:15.361590 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.361576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:15.361994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.361977 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9tlqc\"" Aug 04 10:28:15.364284 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.364271 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.365199 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.365159 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 04 10:28:15.365199 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.365195 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 04 10:28:15.365337 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.365237 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qbtwc\"" Aug 04 10:28:15.365669 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.365654 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 10:28:15.366004 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.365991 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jtmx7\"" Aug 04 10:28:15.366456 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.366437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 10:28:15.366689 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.366583 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 10:28:15.366757 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.366696 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 10:28:15.368720 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.368697 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx"] Aug 04 10:28:15.373932 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.373911 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 10:28:15.374083 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.374068 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p"] Aug 04 10:28:15.375229 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.375191 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-ws6m5"] Aug 04 10:28:15.375936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.375918 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x"] Aug 04 10:28:15.387118 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e94db208-1619-40c0-9b02-9bfb74cb474f-snapshots\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.387195 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94db208-1619-40c0-9b02-9bfb74cb474f-tmp\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.387195 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.387264 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e94db208-1619-40c0-9b02-9bfb74cb474f-service-ca-bundle\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.387264 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/779fd46d-da59-4807-9385-607c2dfda848-serving-cert\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.387333 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387275 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/779fd46d-da59-4807-9385-607c2dfda848-config\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.387333 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glfrq\" (UniqueName: \"kubernetes.io/projected/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-kube-api-access-glfrq\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.387333 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.387417 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzs4\" (UniqueName: \"kubernetes.io/projected/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-kube-api-access-phzs4\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.387513 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e94db208-1619-40c0-9b02-9bfb74cb474f-serving-cert\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.387548 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387540 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e94db208-1619-40c0-9b02-9bfb74cb474f-trusted-ca-bundle\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.387594 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672jg\" (UniqueName: \"kubernetes.io/projected/e94db208-1619-40c0-9b02-9bfb74cb474f-kube-api-access-672jg\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.387628 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87gn\" (UniqueName: \"kubernetes.io/projected/779fd46d-da59-4807-9385-607c2dfda848-kube-api-access-c87gn\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.387628 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb209365-f444-4ec0-99e6-afef201c1ff0-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:15.387771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.387771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:15.387892 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6lf\" (UniqueName: \"kubernetes.io/projected/8bcc4a9d-4c90-472f-9821-8ade3abf9587-kube-api-access-5w6lf\") pod \"network-check-source-65bdbbff45-4cgml\" (UID: \"8bcc4a9d-4c90-472f-9821-8ade3abf9587\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" Aug 04 10:28:15.387892 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.387892 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.387849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.389621 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.389600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.396406 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.396384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzs4\" (UniqueName: \"kubernetes.io/projected/94b88d24-64e8-4d0e-98b4-96c9f17a2d85-kube-api-access-phzs4\") pod \"kube-storage-version-migrator-operator-7878f4b88c-4grhn\" (UID: \"94b88d24-64e8-4d0e-98b4-96c9f17a2d85\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.397153 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.397138 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6lf\" (UniqueName: \"kubernetes.io/projected/8bcc4a9d-4c90-472f-9821-8ade3abf9587-kube-api-access-5w6lf\") pod \"network-check-source-65bdbbff45-4cgml\" (UID: \"8bcc4a9d-4c90-472f-9821-8ade3abf9587\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" Aug 04 10:28:15.474869 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.474806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" Aug 04 10:28:15.480413 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.480392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:15.488816 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.488794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb209365-f444-4ec0-99e6-afef201c1ff0-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:15.488926 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.488892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:15.488986 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.488947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.488986 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.488981 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e94db208-1619-40c0-9b02-9bfb74cb474f-snapshots\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.489082 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94db208-1619-40c0-9b02-9bfb74cb474f-tmp\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.489082 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.489022 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 10:28:15.489177 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.489091 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert podName:fb209365-f444-4ec0-99e6-afef201c1ff0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:15.989072015 +0000 UTC m=+56.940479545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-l7q4x" (UID: "fb209365-f444-4ec0-99e6-afef201c1ff0") : secret "networking-console-plugin-cert" not found Aug 04 10:28:15.489242 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.489182 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:15.489292 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.489248 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls podName:a9bc0030-6e9b-413a-a166-c4d1e9cfd458 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:15.989232162 +0000 UTC m=+56.940639689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-9ht9p" (UID: "a9bc0030-6e9b-413a-a166-c4d1e9cfd458") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:15.489292 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.489508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e94db208-1619-40c0-9b02-9bfb74cb474f-service-ca-bundle\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.489508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94db208-1619-40c0-9b02-9bfb74cb474f-tmp\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.489508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/779fd46d-da59-4807-9385-607c2dfda848-serving-cert\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.489508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/779fd46d-da59-4807-9385-607c2dfda848-config\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.489508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb209365-f444-4ec0-99e6-afef201c1ff0-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:15.489508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glfrq\" (UniqueName: \"kubernetes.io/projected/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-kube-api-access-glfrq\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.489508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e94db208-1619-40c0-9b02-9bfb74cb474f-serving-cert\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.489881 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e94db208-1619-40c0-9b02-9bfb74cb474f-trusted-ca-bundle\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.489881 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672jg\" (UniqueName: \"kubernetes.io/projected/e94db208-1619-40c0-9b02-9bfb74cb474f-kube-api-access-672jg\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.489881 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87gn\" (UniqueName: \"kubernetes.io/projected/779fd46d-da59-4807-9385-607c2dfda848-kube-api-access-c87gn\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.489881 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489617 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e94db208-1619-40c0-9b02-9bfb74cb474f-snapshots\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.490079 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.489977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.490207 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.490186 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/779fd46d-da59-4807-9385-607c2dfda848-config\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.490426 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.490406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e94db208-1619-40c0-9b02-9bfb74cb474f-service-ca-bundle\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.490568 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.490552 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e94db208-1619-40c0-9b02-9bfb74cb474f-trusted-ca-bundle\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.491560 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.491540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/779fd46d-da59-4807-9385-607c2dfda848-serving-cert\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.492020 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.492001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e94db208-1619-40c0-9b02-9bfb74cb474f-serving-cert\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.503151 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.503110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glfrq\" (UniqueName: \"kubernetes.io/projected/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-kube-api-access-glfrq\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.503265 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.503238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-672jg\" (UniqueName: \"kubernetes.io/projected/e94db208-1619-40c0-9b02-9bfb74cb474f-kube-api-access-672jg\") pod \"insights-operator-77696685b-ws6m5\" (UID: \"e94db208-1619-40c0-9b02-9bfb74cb474f\") " pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.503433 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.503398 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87gn\" (UniqueName: \"kubernetes.io/projected/779fd46d-da59-4807-9385-607c2dfda848-kube-api-access-c87gn\") pod \"service-ca-operator-784f97bcf6-mqssx\" (UID: \"779fd46d-da59-4807-9385-607c2dfda848\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.576080 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.576049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" Aug 04 10:28:15.580124 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.580105 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" Aug 04 10:28:15.632754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.632725 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-5kbmf"] Aug 04 10:28:15.636212 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:15.635945 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cbfbcb_c5dd_40dc_9b57_fad40c02bd71.slice/crio-5a49e42c8d2e740a0311a72c6aaac373add129c34ca9678fc31f07ae89f95bb3 WatchSource:0}: Error finding container 5a49e42c8d2e740a0311a72c6aaac373add129c34ca9678fc31f07ae89f95bb3: Status 404 returned error can't find the container with id 5a49e42c8d2e740a0311a72c6aaac373add129c34ca9678fc31f07ae89f95bb3 Aug 04 10:28:15.651569 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.651372 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq"] Aug 04 10:28:15.654179 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:15.654148 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16fa908_9c26_438c_bf68_9e44b999a1fb.slice/crio-acd1a0deb080dff403f66853961b1f4bd4fb66318d2ddcc4dd7ff82d7ef0a572 WatchSource:0}: Error finding container acd1a0deb080dff403f66853961b1f4bd4fb66318d2ddcc4dd7ff82d7ef0a572: Status 404 returned error can't find the container with id acd1a0deb080dff403f66853961b1f4bd4fb66318d2ddcc4dd7ff82d7ef0a572 Aug 04 10:28:15.662875 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.662823 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" Aug 04 10:28:15.683338 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.683273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-ws6m5" Aug 04 10:28:15.737677 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.737646 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml"] Aug 04 10:28:15.742334 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:15.742209 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcc4a9d_4c90_472f_9821_8ade3abf9587.slice/crio-58a68a05495ead897afe51936b0cd93d03225b94edc96f822b2fc6ddfc441acf WatchSource:0}: Error finding container 58a68a05495ead897afe51936b0cd93d03225b94edc96f822b2fc6ddfc441acf: Status 404 returned error can't find the container with id 58a68a05495ead897afe51936b0cd93d03225b94edc96f822b2fc6ddfc441acf Aug 04 10:28:15.757793 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.757723 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn"] Aug 04 10:28:15.761740 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:15.761710 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b88d24_64e8_4d0e_98b4_96c9f17a2d85.slice/crio-50d670e2fb30f50f912f7ddbcbde86a2572a9cd1fb3894c0503a1ed1f9b21a2f WatchSource:0}: Error finding container 50d670e2fb30f50f912f7ddbcbde86a2572a9cd1fb3894c0503a1ed1f9b21a2f: Status 404 returned error can't find the container with id 50d670e2fb30f50f912f7ddbcbde86a2572a9cd1fb3894c0503a1ed1f9b21a2f Aug 04 10:28:15.792582 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.792555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.792711 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.792609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:15.792711 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.792689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:15.792818 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.792706 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:28:15.792818 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.792745 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:28:15.792818 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.792769 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:16.792750661 +0000 UTC m=+57.744158212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : secret "router-metrics-certs-default" not found Aug 04 10:28:15.792818 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.792808 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls podName:bd849d55-f17a-4a9c-bc88-5e60089d83ed nodeName:}" failed. No retries permitted until 2026-08-04 10:28:16.792788807 +0000 UTC m=+57.744196354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-4g2wm" (UID: "bd849d55-f17a-4a9c-bc88-5e60089d83ed") : secret "samples-operator-tls" not found Aug 04 10:28:15.793015 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.792827 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:16.792817689 +0000 UTC m=+57.744225223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : configmap references non-existent config key: service-ca.crt Aug 04 10:28:15.809275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.809247 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx"] Aug 04 10:28:15.812634 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:15.812616 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779fd46d_da59_4807_9385_607c2dfda848.slice/crio-991fff7540573fdcf115897935d99e94b9a9636faa050b131c3c3f98406a26ca WatchSource:0}: Error finding container 991fff7540573fdcf115897935d99e94b9a9636faa050b131c3c3f98406a26ca: Status 404 returned error can't find the container with id 991fff7540573fdcf115897935d99e94b9a9636faa050b131c3c3f98406a26ca Aug 04 10:28:15.816372 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.816317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" event={"ID":"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71","Type":"ContainerStarted","Data":"5a49e42c8d2e740a0311a72c6aaac373add129c34ca9678fc31f07ae89f95bb3"} Aug 04 10:28:15.817366 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.817348 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" event={"ID":"779fd46d-da59-4807-9385-607c2dfda848","Type":"ContainerStarted","Data":"991fff7540573fdcf115897935d99e94b9a9636faa050b131c3c3f98406a26ca"} Aug 04 10:28:15.818450 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.818432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" event={"ID":"94b88d24-64e8-4d0e-98b4-96c9f17a2d85","Type":"ContainerStarted","Data":"50d670e2fb30f50f912f7ddbcbde86a2572a9cd1fb3894c0503a1ed1f9b21a2f"} Aug 04 10:28:15.819916 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.819893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" event={"ID":"8bcc4a9d-4c90-472f-9821-8ade3abf9587","Type":"ContainerStarted","Data":"58a68a05495ead897afe51936b0cd93d03225b94edc96f822b2fc6ddfc441acf"} Aug 04 10:28:15.820868 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.820850 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" event={"ID":"b16fa908-9c26-438c-bf68-9e44b999a1fb","Type":"ContainerStarted","Data":"acd1a0deb080dff403f66853961b1f4bd4fb66318d2ddcc4dd7ff82d7ef0a572"} Aug 04 10:28:15.824323 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.824302 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-ws6m5"] Aug 04 10:28:15.827820 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:15.827799 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94db208_1619_40c0_9b02_9bfb74cb474f.slice/crio-ed411e8d19a7c8ee17a358b6e1036414b8b8e26c8ef47468603352b7107b7c24 WatchSource:0}: Error finding container ed411e8d19a7c8ee17a358b6e1036414b8b8e26c8ef47468603352b7107b7c24: Status 404 returned error can't find the container with id ed411e8d19a7c8ee17a358b6e1036414b8b8e26c8ef47468603352b7107b7c24 Aug 04 10:28:15.994759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.994695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:15.994759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:15.994737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:15.994913 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.994857 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:15.994913 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.994857 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 10:28:15.994913 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.994902 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls podName:a9bc0030-6e9b-413a-a166-c4d1e9cfd458 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:16.994889082 +0000 UTC m=+57.946296609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-9ht9p" (UID: "a9bc0030-6e9b-413a-a166-c4d1e9cfd458") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:15.995004 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:15.994934 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert podName:fb209365-f444-4ec0-99e6-afef201c1ff0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:16.994920924 +0000 UTC m=+57.946328451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-l7q4x" (UID: "fb209365-f444-4ec0-99e6-afef201c1ff0") : secret "networking-console-plugin-cert" not found Aug 04 10:28:16.802960 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:16.802919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:16.803411 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:16.803048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:16.803411 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:16.803104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:16.803411 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:16.803261 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:28:16.803411 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:16.803331 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls podName:bd849d55-f17a-4a9c-bc88-5e60089d83ed nodeName:}" failed. No retries permitted until 2026-08-04 10:28:18.80331271 +0000 UTC m=+59.754720241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-4g2wm" (UID: "bd849d55-f17a-4a9c-bc88-5e60089d83ed") : secret "samples-operator-tls" not found Aug 04 10:28:16.803411 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:16.803405 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:18.803394959 +0000 UTC m=+59.754802490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : configmap references non-existent config key: service-ca.crt Aug 04 10:28:16.803679 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:16.803465 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:28:16.803679 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:16.803498 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:18.803485876 +0000 UTC m=+59.754893407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : secret "router-metrics-certs-default" not found Aug 04 10:28:16.830213 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:16.830182 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-ws6m5" event={"ID":"e94db208-1619-40c0-9b02-9bfb74cb474f","Type":"ContainerStarted","Data":"ed411e8d19a7c8ee17a358b6e1036414b8b8e26c8ef47468603352b7107b7c24"} Aug 04 10:28:17.005757 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:17.005414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:17.005757 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:17.005481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:17.005757 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:17.005588 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 10:28:17.005757 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:17.005657 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert podName:fb209365-f444-4ec0-99e6-afef201c1ff0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:19.005636624 +0000 UTC m=+59.957044174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-l7q4x" (UID: "fb209365-f444-4ec0-99e6-afef201c1ff0") : secret "networking-console-plugin-cert" not found Aug 04 10:28:17.005757 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:17.005663 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:17.005757 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:17.005715 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls podName:a9bc0030-6e9b-413a-a166-c4d1e9cfd458 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:19.005699936 +0000 UTC m=+59.957107463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-9ht9p" (UID: "a9bc0030-6e9b-413a-a166-c4d1e9cfd458") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:17.835412 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:17.835375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" event={"ID":"b16fa908-9c26-438c-bf68-9e44b999a1fb","Type":"ContainerStarted","Data":"dfb899f21f7cff7430ce9773a583fe57340edeff062339a600a83c0e0063c842"} Aug 04 10:28:17.852066 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:17.852014 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-2wdxq" podStartSLOduration=1.50174381 podStartE2EDuration="2.851996842s" totalImagesPullingTime="1.350253032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:17.851212379 +0000 UTC m=+58.802619930" watchObservedRunningTime="2026-08-04 10:28:17.851996842 +0000 UTC m=+58.803404392" Aug 04 10:28:18.823223 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:18.823186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:18.823414 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:18.823293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:18.823414 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:18.823349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:18.823414 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:18.823366 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:22.823343345 +0000 UTC m=+63.774750872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : configmap references non-existent config key: service-ca.crt Aug 04 10:28:18.823584 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:18.823432 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:28:18.823584 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:18.823432 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:28:18.823584 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:18.823482 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls podName:bd849d55-f17a-4a9c-bc88-5e60089d83ed nodeName:}" failed. No retries permitted until 2026-08-04 10:28:22.823466918 +0000 UTC m=+63.774874450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-4g2wm" (UID: "bd849d55-f17a-4a9c-bc88-5e60089d83ed") : secret "samples-operator-tls" not found Aug 04 10:28:18.823584 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:18.823556 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:22.82353699 +0000 UTC m=+63.774944522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : secret "router-metrics-certs-default" not found Aug 04 10:28:19.025389 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:19.025348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:19.025739 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:19.025416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:19.025739 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:19.025497 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 10:28:19.025739 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:19.025561 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:19.025739 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:19.025568 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert podName:fb209365-f444-4ec0-99e6-afef201c1ff0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.02554786 +0000 UTC m=+63.976955405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-l7q4x" (UID: "fb209365-f444-4ec0-99e6-afef201c1ff0") : secret "networking-console-plugin-cert" not found Aug 04 10:28:19.025739 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:19.025628 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls podName:a9bc0030-6e9b-413a-a166-c4d1e9cfd458 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.025614868 +0000 UTC m=+63.977022395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-9ht9p" (UID: "a9bc0030-6e9b-413a-a166-c4d1e9cfd458") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:20.756461 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.756435 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-clkh9" Aug 04 10:28:20.856428 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.856391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" event={"ID":"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71","Type":"ContainerStarted","Data":"ecd1b007f593c358f50f61a26dac200607fdb0ce3e8345551a7e2551dc2445fa"} Aug 04 10:28:20.856818 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.856797 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:20.858822 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.858508 2577 patch_prober.go:28] interesting pod/console-operator-5ccd6b445f-5kbmf container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Aug 04 10:28:20.859083 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.859047 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Aug 04 10:28:20.859688 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.859666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" event={"ID":"779fd46d-da59-4807-9385-607c2dfda848","Type":"ContainerStarted","Data":"5235c980aea41d3da6406ea5f5aa525e05df8d6d977e09323f7a1437e88e2244"} Aug 04 10:28:20.861511 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.861490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" event={"ID":"94b88d24-64e8-4d0e-98b4-96c9f17a2d85","Type":"ContainerStarted","Data":"ac0d320aac81aacb441e5193c4d37599c0b77aa86f96838548d41481fdf6fcf2"} Aug 04 10:28:20.865550 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.865527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" event={"ID":"8bcc4a9d-4c90-472f-9821-8ade3abf9587","Type":"ContainerStarted","Data":"734c127efcf18225b38d837a4964181bc872c5b38662d46df19a00c431232706"} Aug 04 10:28:20.867275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.867156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-ws6m5" event={"ID":"e94db208-1619-40c0-9b02-9bfb74cb474f","Type":"ContainerStarted","Data":"11b47aaaa8ccd522383cf272f38b37cbf13786f6897bfc918836a897efb3d351"} Aug 04 10:28:20.878635 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.878485 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podStartSLOduration=0.818208322 podStartE2EDuration="5.878470471s" totalImagesPullingTime="5.060262149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:20.877156729 +0000 UTC m=+61.828564277" watchObservedRunningTime="2026-08-04 10:28:20.878470471 +0000 UTC m=+61.829878021" Aug 04 10:28:20.893589 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.893545 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-4cgml" podStartSLOduration=0.924446674 podStartE2EDuration="5.893529009s" totalImagesPullingTime="4.969082335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:20.892640003 +0000 UTC m=+61.844047550" watchObservedRunningTime="2026-08-04 10:28:20.893529009 +0000 UTC m=+61.844936559" Aug 04 10:28:20.910319 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.910276 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" podStartSLOduration=0.975007456 podStartE2EDuration="5.910259143s" totalImagesPullingTime="4.935251687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:20.909210125 +0000 UTC m=+61.860617675" watchObservedRunningTime="2026-08-04 10:28:20.910259143 +0000 UTC m=+61.861666697" Aug 04 10:28:20.948573 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.948487 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" podStartSLOduration=1.060431577 podStartE2EDuration="5.948470707s" totalImagesPullingTime="4.88803913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:20.947316621 +0000 UTC m=+61.898724163" watchObservedRunningTime="2026-08-04 10:28:20.948470707 +0000 UTC m=+61.899878257" Aug 04 10:28:20.992607 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:20.992558 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-ws6m5" podStartSLOduration=1.122663805 podStartE2EDuration="5.992541044s" totalImagesPullingTime="4.869877239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:20.991966268 +0000 UTC m=+61.943373818" watchObservedRunningTime="2026-08-04 10:28:20.992541044 +0000 UTC m=+61.943948596" Aug 04 10:28:21.873133 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:21.873105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/0.log" Aug 04 10:28:21.873498 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:21.873146 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" containerID="ecd1b007f593c358f50f61a26dac200607fdb0ce3e8345551a7e2551dc2445fa" exitCode=255 Aug 04 10:28:21.874419 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:21.874067 2577 scope.go:117] "RemoveContainer" containerID="ecd1b007f593c358f50f61a26dac200607fdb0ce3e8345551a7e2551dc2445fa" Aug 04 10:28:21.874419 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:21.874338 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" event={"ID":"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71","Type":"ContainerDied","Data":"ecd1b007f593c358f50f61a26dac200607fdb0ce3e8345551a7e2551dc2445fa"} Aug 04 10:28:22.054093 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.054060 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7cfg9"] Aug 04 10:28:22.057317 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.057295 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.059345 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.059314 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 10:28:22.059455 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.059319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 10:28:22.059873 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.059857 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntlkp\"" Aug 04 10:28:22.069257 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.069236 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7cfg9"] Aug 04 10:28:22.154727 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.154696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjp7\" (UniqueName: \"kubernetes.io/projected/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-api-access-qnjp7\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.154908 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.154733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be213403-b9d8-4d9d-b287-391f07ca66a0-crio-socket\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.154973 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.154927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.155027 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.155009 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.155070 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.155035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be213403-b9d8-4d9d-b287-391f07ca66a0-data-volume\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.256386 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.256344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.256539 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.256446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.256539 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.256473 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:22.256539 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.256472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be213403-b9d8-4d9d-b287-391f07ca66a0-data-volume\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.256657 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.256539 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:22.75652078 +0000 UTC m=+63.707928316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : configmap "kube-rbac-proxy" not found Aug 04 10:28:22.256657 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.256572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjp7\" (UniqueName: \"kubernetes.io/projected/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-api-access-qnjp7\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.256657 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.256596 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:22.256657 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.256600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be213403-b9d8-4d9d-b287-391f07ca66a0-crio-socket\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.256811 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.256662 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:22.75664292 +0000 UTC m=+63.708050451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:22.256811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.256656 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/be213403-b9d8-4d9d-b287-391f07ca66a0-crio-socket\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.256811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.256784 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/be213403-b9d8-4d9d-b287-391f07ca66a0-data-volume\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.268206 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.268179 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjp7\" (UniqueName: \"kubernetes.io/projected/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-api-access-qnjp7\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.760994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.760965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.761152 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.761088 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:22.761152 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.761128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:22.761245 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.761154 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.761136017 +0000 UTC m=+64.712543544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : configmap "kube-rbac-proxy" not found Aug 04 10:28:22.761245 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.761216 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:22.761342 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.761302 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:23.761284763 +0000 UTC m=+64.712692293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:22.862577 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.862544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:22.862732 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.862612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:22.862732 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.862683 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:28:22.862848 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.862741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:30.862725968 +0000 UTC m=+71.814133496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : secret "router-metrics-certs-default" not found Aug 04 10:28:22.862848 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.862750 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 10:28:22.862848 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.862682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:22.862848 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.862766 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:30.862752751 +0000 UTC m=+71.814160279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : configmap references non-existent config key: service-ca.crt Aug 04 10:28:22.862848 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.862805 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls podName:bd849d55-f17a-4a9c-bc88-5e60089d83ed nodeName:}" failed. No retries permitted until 2026-08-04 10:28:30.862788602 +0000 UTC m=+71.814196133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-4g2wm" (UID: "bd849d55-f17a-4a9c-bc88-5e60089d83ed") : secret "samples-operator-tls" not found Aug 04 10:28:22.880012 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.879989 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/1.log" Aug 04 10:28:22.880391 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.880377 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/0.log" Aug 04 10:28:22.880438 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.880409 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" containerID="f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583" exitCode=255 Aug 04 10:28:22.880438 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.880433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" event={"ID":"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71","Type":"ContainerDied","Data":"f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583"} Aug 04 10:28:22.880503 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.880493 2577 scope.go:117] "RemoveContainer" containerID="ecd1b007f593c358f50f61a26dac200607fdb0ce3e8345551a7e2551dc2445fa" Aug 04 10:28:22.880664 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:22.880645 2577 scope.go:117] "RemoveContainer" containerID="f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583" Aug 04 10:28:22.880849 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:22.880805 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-5kbmf_openshift-console-operator(b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" Aug 04 10:28:23.064856 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:23.064777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:23.064856 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:23.064822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:23.065001 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.064911 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 10:28:23.065001 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.064968 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:23.065001 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.064981 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert podName:fb209365-f444-4ec0-99e6-afef201c1ff0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:31.064962754 +0000 UTC m=+72.016370289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-l7q4x" (UID: "fb209365-f444-4ec0-99e6-afef201c1ff0") : secret "networking-console-plugin-cert" not found Aug 04 10:28:23.065097 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.065004 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls podName:a9bc0030-6e9b-413a-a166-c4d1e9cfd458 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:31.064993641 +0000 UTC m=+72.016401192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-9ht9p" (UID: "a9bc0030-6e9b-413a-a166-c4d1e9cfd458") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:23.771733 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:23.771701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:23.771916 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:23.771810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:23.771916 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.771871 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:23.771916 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.771900 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:23.772024 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.771934 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.771919412 +0000 UTC m=+66.723326939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:23.772024 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.771949 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:25.771942751 +0000 UTC m=+66.723350278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : configmap "kube-rbac-proxy" not found Aug 04 10:28:23.885932 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:23.885906 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/1.log" Aug 04 10:28:23.886297 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:23.886229 2577 scope.go:117] "RemoveContainer" containerID="f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583" Aug 04 10:28:23.886385 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:23.886365 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-5kbmf_openshift-console-operator(b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" Aug 04 10:28:24.003543 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:24.003520 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4mkf_40a3ad3e-821e-4f8f-bf56-aeedbf62b949/dns-node-resolver/0.log" Aug 04 10:28:25.005126 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.005067 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bznw4_a7dd768f-e27c-413b-8aeb-fd7c20330835/node-ca/0.log" Aug 04 10:28:25.284419 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.284331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:28:25.284419 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.284392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:28:25.284640 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.284454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:28:25.284640 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.284551 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 10:28:25.284640 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.284564 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 10:28:25.284640 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.284578 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5979db58b7-mr9p5: secret "image-registry-tls" not found Aug 04 10:28:25.284640 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.284611 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls podName:39c508ad-842a-41f1-9fc0-79df29976a48 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:57.28459575 +0000 UTC m=+98.236003278 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls") pod "dns-default-h8frs" (UID: "39c508ad-842a-41f1-9fc0-79df29976a48") : secret "dns-default-metrics-tls" not found Aug 04 10:28:25.284640 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.284624 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls podName:777e50d7-624b-4a82-99c5-56675f43183a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:57.284618761 +0000 UTC m=+98.236026288 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls") pod "image-registry-5979db58b7-mr9p5" (UID: "777e50d7-624b-4a82-99c5-56675f43183a") : secret "image-registry-tls" not found Aug 04 10:28:25.286934 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.286916 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 10:28:25.295143 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.295124 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 10:28:25.295236 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.295166 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs podName:0fa238f6-a750-4b82-8c0c-d31e0fcda574 nodeName:}" failed. No retries permitted until 2026-08-04 10:29:29.295153024 +0000 UTC m=+130.246560552 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs") pod "network-metrics-daemon-xrhhd" (UID: "0fa238f6-a750-4b82-8c0c-d31e0fcda574") : secret "metrics-daemon-secret" not found Aug 04 10:28:25.385278 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.385256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:28:25.385382 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.385334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:28:25.387400 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.387378 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 10:28:25.387877 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.387857 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6zn2\" (UniqueName: \"kubernetes.io/projected/631b0a8d-ae6e-433b-9927-0d0546cce4ee-kube-api-access-v6zn2\") pod \"network-check-target-5gw46\" (UID: \"631b0a8d-ae6e-433b-9927-0d0546cce4ee\") " pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:28:25.398058 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.398035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12828ebf-4f28-42bb-9882-cc4f93bee964-original-pull-secret\") pod \"global-pull-secret-syncer-wdm6f\" (UID: \"12828ebf-4f28-42bb-9882-cc4f93bee964\") " pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:28:25.438148 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.438128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdm6f" Aug 04 10:28:25.452509 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.452493 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-clvkl\"" Aug 04 10:28:25.459368 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.459350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:28:25.481275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.481254 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:25.481592 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.481581 2577 scope.go:117] "RemoveContainer" containerID="f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583" Aug 04 10:28:25.481733 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.481717 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-5kbmf_openshift-console-operator(b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" Aug 04 10:28:25.579460 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.579428 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdm6f"] Aug 04 10:28:25.583523 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:25.583498 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12828ebf_4f28_42bb_9882_cc4f93bee964.slice/crio-5c09a34342bbeb1d31a358ad04a8c37066cc9262d137ff189d282b71ff02ca2e WatchSource:0}: Error finding container 5c09a34342bbeb1d31a358ad04a8c37066cc9262d137ff189d282b71ff02ca2e: Status 404 returned error can't find the container with id 5c09a34342bbeb1d31a358ad04a8c37066cc9262d137ff189d282b71ff02ca2e Aug 04 10:28:25.592022 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.591999 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gw46"] Aug 04 10:28:25.594425 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:25.594402 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631b0a8d_ae6e_433b_9927_0d0546cce4ee.slice/crio-6a5710472d39b7ebc589dc682cf84aaf6be5c18c1a64b4147f00409ec03e9203 WatchSource:0}: Error finding container 6a5710472d39b7ebc589dc682cf84aaf6be5c18c1a64b4147f00409ec03e9203: Status 404 returned error can't find the container with id 6a5710472d39b7ebc589dc682cf84aaf6be5c18c1a64b4147f00409ec03e9203 Aug 04 10:28:25.788125 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.788076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:25.788219 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.788169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:25.788219 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.788210 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:25.788284 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.788246 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:25.788284 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.788262 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:29.788248556 +0000 UTC m=+70.739656084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:25.788284 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:25.788282 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:29.78827126 +0000 UTC m=+70.739678787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : configmap "kube-rbac-proxy" not found Aug 04 10:28:25.802930 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.802911 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rfcm6_16a59ee9-ea69-403e-8ee7-06492cb61c78/serve-healthcheck-canary/0.log" Aug 04 10:28:25.893544 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.893521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdm6f" event={"ID":"12828ebf-4f28-42bb-9882-cc4f93bee964","Type":"ContainerStarted","Data":"5c09a34342bbeb1d31a358ad04a8c37066cc9262d137ff189d282b71ff02ca2e"} Aug 04 10:28:25.894872 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.894845 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gw46" event={"ID":"631b0a8d-ae6e-433b-9927-0d0546cce4ee","Type":"ContainerStarted","Data":"64372255530be10e616373427e15fa9becdc018ebdd5db576f809f0794c2ed5f"} Aug 04 10:28:25.894958 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.894886 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gw46" event={"ID":"631b0a8d-ae6e-433b-9927-0d0546cce4ee","Type":"ContainerStarted","Data":"6a5710472d39b7ebc589dc682cf84aaf6be5c18c1a64b4147f00409ec03e9203"} Aug 04 10:28:25.894997 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.894962 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:28:25.920316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:25.917237 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5gw46" podStartSLOduration=66.917219689 podStartE2EDuration="1m6.917219689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:25.916394578 +0000 UTC m=+66.867802127" watchObservedRunningTime="2026-08-04 10:28:25.917219689 +0000 UTC m=+66.868627238" Aug 04 10:28:29.824561 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:29.824531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:29.824958 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:29.824590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:29.824958 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:29.824657 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 10:28:29.824958 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:29.824726 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:37.824708591 +0000 UTC m=+78.776116141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : configmap "kube-rbac-proxy" not found Aug 04 10:28:29.824958 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:29.824731 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:29.824958 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:29.824770 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:37.82475833 +0000 UTC m=+78.776165857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:29.909717 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:29.909688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdm6f" event={"ID":"12828ebf-4f28-42bb-9882-cc4f93bee964","Type":"ContainerStarted","Data":"30adb5ff9b61a51f3ed8c1a5fc2310cb7c3ae700db7fa0926e489bbffd8b06fc"} Aug 04 10:28:29.929026 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:29.928985 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wdm6f" podStartSLOduration=66.118131731 podStartE2EDuration="1m9.928971197s" totalImagesPullingTime="3.810839466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:29.928349712 +0000 UTC m=+70.879757252" watchObservedRunningTime="2026-08-04 10:28:29.928971197 +0000 UTC m=+70.880378746" Aug 04 10:28:30.857356 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:30.857322 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:30.857718 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:30.857678 2577 scope.go:117] "RemoveContainer" containerID="f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583" Aug 04 10:28:30.857845 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:30.857814 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-5kbmf_openshift-console-operator(b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" Aug 04 10:28:30.933011 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:30.932984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:30.933165 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:30.933073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:30.933165 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:30.933130 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 10:28:30.933291 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:30.933185 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:46.933167137 +0000 UTC m=+87.884574675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : secret "router-metrics-certs-default" not found Aug 04 10:28:30.933361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:30.933286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:30.933437 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:30.933420 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle podName:23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a nodeName:}" failed. No retries permitted until 2026-08-04 10:28:46.933404267 +0000 UTC m=+87.884811795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle") pod "router-default-587b49775d-8b8rl" (UID: "23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a") : configmap references non-existent config key: service-ca.crt Aug 04 10:28:30.935289 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:30.935267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd849d55-f17a-4a9c-bc88-5e60089d83ed-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-4g2wm\" (UID: \"bd849d55-f17a-4a9c-bc88-5e60089d83ed\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:31.089535 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:31.089507 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nbfr5\"" Aug 04 10:28:31.096786 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:31.096763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" Aug 04 10:28:31.135091 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:31.135031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:31.135091 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:31.135086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:31.135233 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:31.135183 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Aug 04 10:28:31.135267 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:31.135238 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert podName:fb209365-f444-4ec0-99e6-afef201c1ff0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:47.135225423 +0000 UTC m=+88.086632955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-l7q4x" (UID: "fb209365-f444-4ec0-99e6-afef201c1ff0") : secret "networking-console-plugin-cert" not found Aug 04 10:28:31.135267 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:31.135183 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:31.135340 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:31.135302 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls podName:a9bc0030-6e9b-413a-a166-c4d1e9cfd458 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:47.13529127 +0000 UTC m=+88.086698800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-9ht9p" (UID: "a9bc0030-6e9b-413a-a166-c4d1e9cfd458") : secret "cluster-monitoring-operator-tls" not found Aug 04 10:28:31.221311 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:31.219339 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm"] Aug 04 10:28:31.918495 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:31.918462 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" event={"ID":"bd849d55-f17a-4a9c-bc88-5e60089d83ed","Type":"ContainerStarted","Data":"02e2842097e77d5b130dfc916f46dd2e55e296d8d2ee67c5fa803edf6f7e1c1e"} Aug 04 10:28:33.926850 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:33.926801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" event={"ID":"bd849d55-f17a-4a9c-bc88-5e60089d83ed","Type":"ContainerStarted","Data":"70586f99ee931d7a2b51c1d0e9d5d152bfb320db29dc903956558917441d405c"} Aug 04 10:28:33.927198 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:33.926868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" event={"ID":"bd849d55-f17a-4a9c-bc88-5e60089d83ed","Type":"ContainerStarted","Data":"8dc82dc733741e2ba3055b8c960ec0b0c63e7f028dadcdbed9e03c3419f8ebc0"} Aug 04 10:28:33.956097 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:33.956051 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-4g2wm" podStartSLOduration=16.850847569 podStartE2EDuration="18.956035706s" totalImagesPullingTime="2.105188137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:33.955489241 +0000 UTC m=+74.906896791" watchObservedRunningTime="2026-08-04 10:28:33.956035706 +0000 UTC m=+74.907443233" Aug 04 10:28:37.890580 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:37.890545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:37.890966 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:37.890608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:37.890966 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:37.890702 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 10:28:37.890966 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:37.890778 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls podName:be213403-b9d8-4d9d-b287-391f07ca66a0 nodeName:}" failed. No retries permitted until 2026-08-04 10:28:53.890758289 +0000 UTC m=+94.842165824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cfg9" (UID: "be213403-b9d8-4d9d-b287-391f07ca66a0") : secret "insights-runtime-extractor-tls" not found Aug 04 10:28:37.891188 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:37.891166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/be213403-b9d8-4d9d-b287-391f07ca66a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:43.526793 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:43.526658 2577 scope.go:117] "RemoveContainer" containerID="f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583" Aug 04 10:28:43.962152 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:43.962086 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:28:43.962503 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:43.962489 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/1.log" Aug 04 10:28:43.962558 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:43.962521 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" containerID="a255d8353b44b6a18c4d4894c4bdc3a5401c8066352df81b332aedccd11fe2b5" exitCode=255 Aug 04 10:28:43.962597 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:43.962553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" event={"ID":"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71","Type":"ContainerDied","Data":"a255d8353b44b6a18c4d4894c4bdc3a5401c8066352df81b332aedccd11fe2b5"} Aug 04 10:28:43.962630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:43.962597 2577 scope.go:117] "RemoveContainer" containerID="f74c1d84d837a8c6a42e1a91b681ce18a3f1337f89c346b8c36544bbcf093583" Aug 04 10:28:43.962907 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:43.962889 2577 scope.go:117] "RemoveContainer" containerID="a255d8353b44b6a18c4d4894c4bdc3a5401c8066352df81b332aedccd11fe2b5" Aug 04 10:28:43.963074 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:43.963055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-5kbmf_openshift-console-operator(b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" Aug 04 10:28:44.967384 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:44.967345 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:28:45.480637 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:45.480606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:45.480921 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:45.480908 2577 scope.go:117] "RemoveContainer" containerID="a255d8353b44b6a18c4d4894c4bdc3a5401c8066352df81b332aedccd11fe2b5" Aug 04 10:28:45.481085 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:45.481066 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-5kbmf_openshift-console-operator(b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" Aug 04 10:28:46.963024 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:46.962990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:46.963581 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:46.963071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:46.963642 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:46.963612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-service-ca-bundle\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:46.965382 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:46.965359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a-metrics-certs\") pod \"router-default-587b49775d-8b8rl\" (UID: \"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a\") " pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:46.968640 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:46.968625 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wxfrj\"" Aug 04 10:28:46.977111 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:46.977095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:47.111106 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.111082 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-587b49775d-8b8rl"] Aug 04 10:28:47.113670 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:47.113646 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dc1b44_3c3f_4aa3_a9ad_0a95f697b70a.slice/crio-5351b609ce4e09e2c00e3dd5f82637052eb5eb700e03a0fdf4dfb27446761fd4 WatchSource:0}: Error finding container 5351b609ce4e09e2c00e3dd5f82637052eb5eb700e03a0fdf4dfb27446761fd4: Status 404 returned error can't find the container with id 5351b609ce4e09e2c00e3dd5f82637052eb5eb700e03a0fdf4dfb27446761fd4 Aug 04 10:28:47.165327 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.165303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:47.165459 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.165343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:47.167541 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.167516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb209365-f444-4ec0-99e6-afef201c1ff0-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-l7q4x\" (UID: \"fb209365-f444-4ec0-99e6-afef201c1ff0\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:47.167627 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.167562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9bc0030-6e9b-413a-a166-c4d1e9cfd458-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-9ht9p\" (UID: \"a9bc0030-6e9b-413a-a166-c4d1e9cfd458\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:47.172565 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.172547 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9tlqc\"" Aug 04 10:28:47.180494 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.180476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" Aug 04 10:28:47.180754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.180739 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qbtwc\"" Aug 04 10:28:47.189358 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.189337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" Aug 04 10:28:47.319903 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.319864 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p"] Aug 04 10:28:47.323085 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:47.323059 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bc0030_6e9b_413a_a166_c4d1e9cfd458.slice/crio-01801e70c04d2d12deac3e3615f6dd1de6aec1e519c69ff16dfa6a139007b34c WatchSource:0}: Error finding container 01801e70c04d2d12deac3e3615f6dd1de6aec1e519c69ff16dfa6a139007b34c: Status 404 returned error can't find the container with id 01801e70c04d2d12deac3e3615f6dd1de6aec1e519c69ff16dfa6a139007b34c Aug 04 10:28:47.339630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.339600 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x"] Aug 04 10:28:47.343765 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:47.343735 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb209365_f444_4ec0_99e6_afef201c1ff0.slice/crio-2788e40951dd289c4433b72734f4e9b18311435f747918b8f7b7a462c3252cd8 WatchSource:0}: Error finding container 2788e40951dd289c4433b72734f4e9b18311435f747918b8f7b7a462c3252cd8: Status 404 returned error can't find the container with id 2788e40951dd289c4433b72734f4e9b18311435f747918b8f7b7a462c3252cd8 Aug 04 10:28:47.979853 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.979777 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-587b49775d-8b8rl" event={"ID":"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a","Type":"ContainerStarted","Data":"44ad87141031371e541acc5106ae33e074ff17f29594d5cafd9d4762ed4c5e44"} Aug 04 10:28:47.980333 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.979857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-587b49775d-8b8rl" event={"ID":"23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a","Type":"ContainerStarted","Data":"5351b609ce4e09e2c00e3dd5f82637052eb5eb700e03a0fdf4dfb27446761fd4"} Aug 04 10:28:47.981203 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.981172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" event={"ID":"fb209365-f444-4ec0-99e6-afef201c1ff0","Type":"ContainerStarted","Data":"2788e40951dd289c4433b72734f4e9b18311435f747918b8f7b7a462c3252cd8"} Aug 04 10:28:47.982411 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:47.982389 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" event={"ID":"a9bc0030-6e9b-413a-a166-c4d1e9cfd458","Type":"ContainerStarted","Data":"01801e70c04d2d12deac3e3615f6dd1de6aec1e519c69ff16dfa6a139007b34c"} Aug 04 10:28:48.007375 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:48.007318 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-587b49775d-8b8rl" podStartSLOduration=33.007300745 podStartE2EDuration="33.007300745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:48.006124051 +0000 UTC m=+88.957531598" watchObservedRunningTime="2026-08-04 10:28:48.007300745 +0000 UTC m=+88.958708293" Aug 04 10:28:48.978366 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:48.978286 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:48.981059 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:48.981036 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:48.987897 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:48.987867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" event={"ID":"fb209365-f444-4ec0-99e6-afef201c1ff0","Type":"ContainerStarted","Data":"0ff4c0b24d17bc6e4d9ea880316232ef334d4ee42a98e0622b38cc9938899ae2"} Aug 04 10:28:48.988005 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:48.987942 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:48.989141 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:48.989119 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-587b49775d-8b8rl" Aug 04 10:28:49.054461 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:49.054409 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-l7q4x" podStartSLOduration=32.88314278 podStartE2EDuration="34.054389976s" totalImagesPullingTime="1.171247196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:49.052637169 +0000 UTC m=+90.004044720" watchObservedRunningTime="2026-08-04 10:28:49.054389976 +0000 UTC m=+90.005797526" Aug 04 10:28:49.992890 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:49.992850 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" event={"ID":"a9bc0030-6e9b-413a-a166-c4d1e9cfd458","Type":"ContainerStarted","Data":"bba0647040af7cae924ebbf3a044ad84cd3802aaeeabcc5296896ea39b6736fc"} Aug 04 10:28:50.857350 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:50.857318 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:28:50.857706 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:50.857693 2577 scope.go:117] "RemoveContainer" containerID="a255d8353b44b6a18c4d4894c4bdc3a5401c8066352df81b332aedccd11fe2b5" Aug 04 10:28:50.857893 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:28:50.857875 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-5kbmf_openshift-console-operator(b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" podUID="b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71" Aug 04 10:28:53.919669 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:53.919632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:53.921816 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:53.921795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/be213403-b9d8-4d9d-b287-391f07ca66a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cfg9\" (UID: \"be213403-b9d8-4d9d-b287-391f07ca66a0\") " pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:54.166893 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:54.166870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7cfg9" Aug 04 10:28:54.336627 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:54.336566 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-9ht9p" podStartSLOduration=37.28274718 podStartE2EDuration="39.336545866s" totalImagesPullingTime="2.053798686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:50.012619915 +0000 UTC m=+90.964027464" watchObservedRunningTime="2026-08-04 10:28:54.336545866 +0000 UTC m=+95.287953417" Aug 04 10:28:54.337134 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:54.337114 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7cfg9"] Aug 04 10:28:54.340530 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:54.340503 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe213403_b9d8_4d9d_b287_391f07ca66a0.slice/crio-f2f3d6e15229325cb547b6cc11e3fa19027c68cb9955a6e2cb4a9ed0b57c67c8 WatchSource:0}: Error finding container f2f3d6e15229325cb547b6cc11e3fa19027c68cb9955a6e2cb4a9ed0b57c67c8: Status 404 returned error can't find the container with id f2f3d6e15229325cb547b6cc11e3fa19027c68cb9955a6e2cb4a9ed0b57c67c8 Aug 04 10:28:55.012229 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:55.012193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfg9" event={"ID":"be213403-b9d8-4d9d-b287-391f07ca66a0","Type":"ContainerStarted","Data":"650fb18659a2687d74dccd3fb704bcd1824c3850993632b3dbf964d9b52c3daa"} Aug 04 10:28:55.012569 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:55.012267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfg9" event={"ID":"be213403-b9d8-4d9d-b287-391f07ca66a0","Type":"ContainerStarted","Data":"f2f3d6e15229325cb547b6cc11e3fa19027c68cb9955a6e2cb4a9ed0b57c67c8"} Aug 04 10:28:56.018290 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:56.018252 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfg9" event={"ID":"be213403-b9d8-4d9d-b287-391f07ca66a0","Type":"ContainerStarted","Data":"8bbf5eb99710bcc162dbb7c50b52bcc24e6847b5dc89eaf4d8fdf4ea0a8e1f41"} Aug 04 10:28:56.900621 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:56.900589 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5gw46" Aug 04 10:28:57.023189 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.023161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cfg9" event={"ID":"be213403-b9d8-4d9d-b287-391f07ca66a0","Type":"ContainerStarted","Data":"0ae44aef5fdea7fe274acaad5a247f5bef8bc18581bd2ffcbe7e83a9dbd3f5b5"} Aug 04 10:28:57.345147 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.345070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:28:57.345147 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.345134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:28:57.347318 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.347286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/39c508ad-842a-41f1-9fc0-79df29976a48-metrics-tls\") pod \"dns-default-h8frs\" (UID: \"39c508ad-842a-41f1-9fc0-79df29976a48\") " pod="openshift-dns/dns-default-h8frs" Aug 04 10:28:57.347457 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.347385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"image-registry-5979db58b7-mr9p5\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:28:57.612982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.612914 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wt4lh\"" Aug 04 10:28:57.621703 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.621681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:28:57.622715 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.622693 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wlm2z\"" Aug 04 10:28:57.631543 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.631526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h8frs" Aug 04 10:28:57.755406 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.755352 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7cfg9" podStartSLOduration=33.573564595 podStartE2EDuration="35.755334725s" totalImagesPullingTime="2.18177013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 10:28:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:57.043912803 +0000 UTC m=+97.995320351" watchObservedRunningTime="2026-08-04 10:28:57.755334725 +0000 UTC m=+98.706742273" Aug 04 10:28:57.756184 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.756161 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5979db58b7-mr9p5"] Aug 04 10:28:57.760326 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:57.760301 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777e50d7_624b_4a82_99c5_56675f43183a.slice/crio-3af84a67a326a70a5227d68f57e1c70aa17b8dd2e424f37aae6bbe4657188fa2 WatchSource:0}: Error finding container 3af84a67a326a70a5227d68f57e1c70aa17b8dd2e424f37aae6bbe4657188fa2: Status 404 returned error can't find the container with id 3af84a67a326a70a5227d68f57e1c70aa17b8dd2e424f37aae6bbe4657188fa2 Aug 04 10:28:57.771702 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:57.771680 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h8frs"] Aug 04 10:28:57.775168 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:28:57.775140 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c508ad_842a_41f1_9fc0_79df29976a48.slice/crio-8cfcf20d128ddeb1be9e48b1cbef8928040ca31fbaa3465b0c417802baf19170 WatchSource:0}: Error finding container 8cfcf20d128ddeb1be9e48b1cbef8928040ca31fbaa3465b0c417802baf19170: Status 404 returned error can't find the container with id 8cfcf20d128ddeb1be9e48b1cbef8928040ca31fbaa3465b0c417802baf19170 Aug 04 10:28:58.028186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:58.028154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" event={"ID":"777e50d7-624b-4a82-99c5-56675f43183a","Type":"ContainerStarted","Data":"8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf"} Aug 04 10:28:58.028603 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:58.028201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" event={"ID":"777e50d7-624b-4a82-99c5-56675f43183a","Type":"ContainerStarted","Data":"3af84a67a326a70a5227d68f57e1c70aa17b8dd2e424f37aae6bbe4657188fa2"} Aug 04 10:28:58.028603 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:58.028267 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:28:58.029370 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:58.029341 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8frs" event={"ID":"39c508ad-842a-41f1-9fc0-79df29976a48","Type":"ContainerStarted","Data":"8cfcf20d128ddeb1be9e48b1cbef8928040ca31fbaa3465b0c417802baf19170"} Aug 04 10:28:58.049488 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:28:58.049441 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" podStartSLOduration=98.049424241 podStartE2EDuration="1m38.049424241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:28:58.04880035 +0000 UTC m=+99.000207905" watchObservedRunningTime="2026-08-04 10:28:58.049424241 +0000 UTC m=+99.000831790" Aug 04 10:29:00.037319 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:00.037285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8frs" event={"ID":"39c508ad-842a-41f1-9fc0-79df29976a48","Type":"ContainerStarted","Data":"3d08deeab40e62c3c945a3b5f84d88f670a736fd7d35580f580985c5f815108f"} Aug 04 10:29:00.037674 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:00.037337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8frs" event={"ID":"39c508ad-842a-41f1-9fc0-79df29976a48","Type":"ContainerStarted","Data":"f7374825f34418f06b17455c024703cca1f003b3daedff44f32d79165a70d164"} Aug 04 10:29:00.037674 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:00.037424 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h8frs" Aug 04 10:29:00.056646 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:00.056593 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h8frs" podStartSLOduration=65.408017594 podStartE2EDuration="1m7.056576692s" totalImagesPullingTime="1.648559098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:00.05490411 +0000 UTC m=+101.006311659" watchObservedRunningTime="2026-08-04 10:29:00.056576692 +0000 UTC m=+101.007984243" Aug 04 10:29:01.364479 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.364450 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp"] Aug 04 10:29:01.397654 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.397625 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp"] Aug 04 10:29:01.397789 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.397678 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nm65z"] Aug 04 10:29:01.397789 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.397768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.399713 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.399689 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 10:29:01.400041 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.400020 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qt7j8\"" Aug 04 10:29:01.400164 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.400056 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 10:29:01.400164 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.400093 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 10:29:01.411483 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.411466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.413512 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.413496 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rg4dt\"" Aug 04 10:29:01.413785 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.413767 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 10:29:01.414132 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.413991 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 10:29:01.414319 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.414304 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 10:29:01.578747 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7jr\" (UniqueName: \"kubernetes.io/projected/885a2f93-5a78-448f-ad26-a79a716fc56c-kube-api-access-gk7jr\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.578747 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-wtmp\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.578977 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-tls\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.578977 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.578977 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885a2f93-5a78-448f-ad26-a79a716fc56c-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.578977 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578900 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.578977 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.578977 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.578950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-accelerators-collector-config\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.579239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.579029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmsx\" (UniqueName: \"kubernetes.io/projected/209f6e41-5a93-4e31-ac31-03a3d3e0d068-kube-api-access-ncmsx\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.579239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.579079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-sys\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.579239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.579105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-textfile\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.579239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.579140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-root\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.579239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.579174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/209f6e41-5a93-4e31-ac31-03a3d3e0d068-metrics-client-ca\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.679787 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-accelerators-collector-config\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.679787 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmsx\" (UniqueName: \"kubernetes.io/projected/209f6e41-5a93-4e31-ac31-03a3d3e0d068-kube-api-access-ncmsx\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-sys\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-textfile\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-root\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/209f6e41-5a93-4e31-ac31-03a3d3e0d068-metrics-client-ca\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679898 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-sys\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-root\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7jr\" (UniqueName: \"kubernetes.io/projected/885a2f93-5a78-448f-ad26-a79a716fc56c-kube-api-access-gk7jr\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.680008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.679997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-wtmp\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-tls\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885a2f93-5a78-448f-ad26-a79a716fc56c-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-wtmp\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680189 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-textfile\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680361 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:29:01.680266 2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Aug 04 10:29:01.680656 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:29:01.680375 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-tls podName:885a2f93-5a78-448f-ad26-a79a716fc56c nodeName:}" failed. No retries permitted until 2026-08-04 10:29:02.180355679 +0000 UTC m=+103.131763206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-wqvtp" (UID: "885a2f93-5a78-448f-ad26-a79a716fc56c") : secret "openshift-state-metrics-tls" not found Aug 04 10:29:01.680656 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680378 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-accelerators-collector-config\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.680656 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.680479 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/209f6e41-5a93-4e31-ac31-03a3d3e0d068-metrics-client-ca\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.681027 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.681003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/885a2f93-5a78-448f-ad26-a79a716fc56c-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.682358 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.682338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-tls\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.682358 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.682350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/209f6e41-5a93-4e31-ac31-03a3d3e0d068-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.682527 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.682500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.690292 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.690267 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmsx\" (UniqueName: \"kubernetes.io/projected/209f6e41-5a93-4e31-ac31-03a3d3e0d068-kube-api-access-ncmsx\") pod \"node-exporter-nm65z\" (UID: \"209f6e41-5a93-4e31-ac31-03a3d3e0d068\") " pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.695321 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.690428 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7jr\" (UniqueName: \"kubernetes.io/projected/885a2f93-5a78-448f-ad26-a79a716fc56c-kube-api-access-gk7jr\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:01.719354 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:01.719331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nm65z" Aug 04 10:29:01.761730 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:29:01.761702 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209f6e41_5a93_4e31_ac31_03a3d3e0d068.slice/crio-30efd5d20833020f833d8480a615b8cc6f480ef6306c36fd7fe6754c5d3d513f WatchSource:0}: Error finding container 30efd5d20833020f833d8480a615b8cc6f480ef6306c36fd7fe6754c5d3d513f: Status 404 returned error can't find the container with id 30efd5d20833020f833d8480a615b8cc6f480ef6306c36fd7fe6754c5d3d513f Aug 04 10:29:02.047119 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:02.047060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nm65z" event={"ID":"209f6e41-5a93-4e31-ac31-03a3d3e0d068","Type":"ContainerStarted","Data":"30efd5d20833020f833d8480a615b8cc6f480ef6306c36fd7fe6754c5d3d513f"} Aug 04 10:29:02.184384 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:02.184354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:02.186740 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:02.186716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/885a2f93-5a78-448f-ad26-a79a716fc56c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wqvtp\" (UID: \"885a2f93-5a78-448f-ad26-a79a716fc56c\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:02.307058 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:02.306987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" Aug 04 10:29:02.458198 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:02.458173 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp"] Aug 04 10:29:02.461052 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:29:02.461015 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885a2f93_5a78_448f_ad26_a79a716fc56c.slice/crio-2a047cbc95a9ed279d1ea89443afe04ad89e8b653094cc4aa14398e19ab090d6 WatchSource:0}: Error finding container 2a047cbc95a9ed279d1ea89443afe04ad89e8b653094cc4aa14398e19ab090d6: Status 404 returned error can't find the container with id 2a047cbc95a9ed279d1ea89443afe04ad89e8b653094cc4aa14398e19ab090d6 Aug 04 10:29:03.052190 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.052161 2577 generic.go:356] "Generic (PLEG): container finished" podID="209f6e41-5a93-4e31-ac31-03a3d3e0d068" containerID="5570c09beb65c12a867706361c2f720d8dd5aececfffc27a8d787ef1103ccdb9" exitCode=0 Aug 04 10:29:03.052300 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.052240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nm65z" event={"ID":"209f6e41-5a93-4e31-ac31-03a3d3e0d068","Type":"ContainerDied","Data":"5570c09beb65c12a867706361c2f720d8dd5aececfffc27a8d787ef1103ccdb9"} Aug 04 10:29:03.054079 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.054059 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" event={"ID":"885a2f93-5a78-448f-ad26-a79a716fc56c","Type":"ContainerStarted","Data":"9377960df4c514b1a40da0931e383edc45afe971139f73ffc7262241442b7485"} Aug 04 10:29:03.054151 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.054101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" event={"ID":"885a2f93-5a78-448f-ad26-a79a716fc56c","Type":"ContainerStarted","Data":"6bd3f6a35aa54fba40c5ff1369c939a7981b686405ff064f009d1d860d150d1f"} Aug 04 10:29:03.054151 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.054119 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" event={"ID":"885a2f93-5a78-448f-ad26-a79a716fc56c","Type":"ContainerStarted","Data":"2a047cbc95a9ed279d1ea89443afe04ad89e8b653094cc4aa14398e19ab090d6"} Aug 04 10:29:03.506129 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.506101 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bd5675968-grpgj"] Aug 04 10:29:03.510387 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.510366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.512660 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.512634 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 04 10:29:03.512778 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.512671 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 04 10:29:03.512992 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.512973 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hjbvb\"" Aug 04 10:29:03.513081 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.513063 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 04 10:29:03.513145 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.513119 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 04 10:29:03.513326 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.513310 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7pfoc7v43qvui\"" Aug 04 10:29:03.513326 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.513322 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 04 10:29:03.524651 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.524627 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bd5675968-grpgj"] Aug 04 10:29:03.695746 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.695724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-metrics-client-ca\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.695841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.695765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.695885 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.695825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.695885 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.695861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wh8\" (UniqueName: \"kubernetes.io/projected/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-kube-api-access-g2wh8\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.695957 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.695922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-tls\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.696001 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.695971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.696036 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.696004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.696036 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.696031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-grpc-tls\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797325 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-metrics-client-ca\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797325 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wh8\" (UniqueName: \"kubernetes.io/projected/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-kube-api-access-g2wh8\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797395 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-tls\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.797521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.797453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-grpc-tls\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.798078 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.798021 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-metrics-client-ca\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.800066 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.800035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-tls\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.800300 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.800276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-grpc-tls\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.800375 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.800339 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.800423 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.800377 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.800465 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.800420 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.800465 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.800436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.805025 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.805008 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wh8\" (UniqueName: \"kubernetes.io/projected/bb5ce1e9-daf7-488c-aca8-23c3a12bde78-kube-api-access-g2wh8\") pod \"thanos-querier-7bd5675968-grpgj\" (UID: \"bb5ce1e9-daf7-488c-aca8-23c3a12bde78\") " pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.820899 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.820883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:03.950560 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:03.950463 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bd5675968-grpgj"] Aug 04 10:29:03.953819 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:29:03.953792 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5ce1e9_daf7_488c_aca8_23c3a12bde78.slice/crio-a46980f4564984e4856a522172b2ac4530dc28d7102e74ff338bdabd073dcee9 WatchSource:0}: Error finding container a46980f4564984e4856a522172b2ac4530dc28d7102e74ff338bdabd073dcee9: Status 404 returned error can't find the container with id a46980f4564984e4856a522172b2ac4530dc28d7102e74ff338bdabd073dcee9 Aug 04 10:29:04.061009 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:04.060944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" event={"ID":"bb5ce1e9-daf7-488c-aca8-23c3a12bde78","Type":"ContainerStarted","Data":"a46980f4564984e4856a522172b2ac4530dc28d7102e74ff338bdabd073dcee9"} Aug 04 10:29:04.062751 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:04.062728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nm65z" event={"ID":"209f6e41-5a93-4e31-ac31-03a3d3e0d068","Type":"ContainerStarted","Data":"b8ea3658d06a57ca3ab35bdc33ead6eebd49a66db01db8438eb4d327c7e31cb0"} Aug 04 10:29:04.062861 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:04.062768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nm65z" event={"ID":"209f6e41-5a93-4e31-ac31-03a3d3e0d068","Type":"ContainerStarted","Data":"f5921249e85973b7a580efac028e4fd0def4e717e608ed3766ce0ca788550b70"} Aug 04 10:29:04.064660 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:04.064641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" event={"ID":"885a2f93-5a78-448f-ad26-a79a716fc56c","Type":"ContainerStarted","Data":"fd145a2f9562d9e4c253712b190271502ee2689924d702cc7ec16ba62f266e8f"} Aug 04 10:29:04.090395 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:04.090353 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nm65z" podStartSLOduration=1.904156612 podStartE2EDuration="3.090337995s" totalImagesPullingTime="1.126830165s" totalInitContainerRuntime="59.351218ms" isStatefulPod=true podCreationTimestamp="2026-08-04 10:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:04.09016045 +0000 UTC m=+105.041568001" watchObservedRunningTime="2026-08-04 10:29:04.090337995 +0000 UTC m=+105.041745540" Aug 04 10:29:04.121066 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:04.121022 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wqvtp" podStartSLOduration=2.041364627 podStartE2EDuration="3.120962652s" totalImagesPullingTime="1.079598025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:04.119432577 +0000 UTC m=+105.070840124" watchObservedRunningTime="2026-08-04 10:29:04.120962652 +0000 UTC m=+105.072370202" Aug 04 10:29:05.526581 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:05.526548 2577 scope.go:117] "RemoveContainer" containerID="a255d8353b44b6a18c4d4894c4bdc3a5401c8066352df81b332aedccd11fe2b5" Aug 04 10:29:06.074985 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.074946 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:29:06.075108 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.075080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" event={"ID":"b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71","Type":"ContainerStarted","Data":"c55007f1e7a4fbc11949d3532040ae9dcb2a9fab385c85d2b77634315018526b"} Aug 04 10:29:06.075543 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.075509 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:29:06.077411 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.077390 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" event={"ID":"bb5ce1e9-daf7-488c-aca8-23c3a12bde78","Type":"ContainerStarted","Data":"305fc6d801be7c673202d7cd73c704c550946282bf95ae787756dbbc20a2f8c2"} Aug 04 10:29:06.077512 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.077438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" event={"ID":"bb5ce1e9-daf7-488c-aca8-23c3a12bde78","Type":"ContainerStarted","Data":"51117e5aa9629f21e92a338d162adb6672931155df1f27933bf97a58dca58a14"} Aug 04 10:29:06.077512 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.077468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" event={"ID":"bb5ce1e9-daf7-488c-aca8-23c3a12bde78","Type":"ContainerStarted","Data":"1661d011c1a4d80047ae10641e409fe8bb96e638b0a332a5445043ecbaa813ac"} Aug 04 10:29:06.182283 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.182255 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5"] Aug 04 10:29:06.185429 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.185414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.187568 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.187543 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 10:29:06.187568 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.187557 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vnsgs\"" Aug 04 10:29:06.203090 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.203067 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5"] Aug 04 10:29:06.214635 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.214614 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f225b1e8-2972-4b59-b10a-4d684718e516-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-h2hf5\" (UID: \"f225b1e8-2972-4b59-b10a-4d684718e516\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.214739 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.214641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mvtj\" (UniqueName: \"kubernetes.io/projected/f225b1e8-2972-4b59-b10a-4d684718e516-kube-api-access-9mvtj\") pod \"monitoring-plugin-6dffbbd67-h2hf5\" (UID: \"f225b1e8-2972-4b59-b10a-4d684718e516\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.216025 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.216005 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-5kbmf" Aug 04 10:29:06.315620 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.315550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f225b1e8-2972-4b59-b10a-4d684718e516-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-h2hf5\" (UID: \"f225b1e8-2972-4b59-b10a-4d684718e516\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.315620 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.315587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mvtj\" (UniqueName: \"kubernetes.io/projected/f225b1e8-2972-4b59-b10a-4d684718e516-kube-api-access-9mvtj\") pod \"monitoring-plugin-6dffbbd67-h2hf5\" (UID: \"f225b1e8-2972-4b59-b10a-4d684718e516\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.317781 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.317755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f225b1e8-2972-4b59-b10a-4d684718e516-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-h2hf5\" (UID: \"f225b1e8-2972-4b59-b10a-4d684718e516\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.325157 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.325138 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mvtj\" (UniqueName: \"kubernetes.io/projected/f225b1e8-2972-4b59-b10a-4d684718e516-kube-api-access-9mvtj\") pod \"monitoring-plugin-6dffbbd67-h2hf5\" (UID: \"f225b1e8-2972-4b59-b10a-4d684718e516\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.494228 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.494200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:06.616662 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.616631 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5"] Aug 04 10:29:06.620557 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:29:06.620530 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf225b1e8_2972_4b59_b10a_4d684718e516.slice/crio-b86fcf2b2bf77611e1a84b04fa2a23510e8f385971fa5bf07b510964fddea07a WatchSource:0}: Error finding container b86fcf2b2bf77611e1a84b04fa2a23510e8f385971fa5bf07b510964fddea07a: Status 404 returned error can't find the container with id b86fcf2b2bf77611e1a84b04fa2a23510e8f385971fa5bf07b510964fddea07a Aug 04 10:29:06.647332 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.647306 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9"] Aug 04 10:29:06.650827 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.650811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.653055 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.653034 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 04 10:29:06.653158 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.653124 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 04 10:29:06.653246 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.653038 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g6z4c\"" Aug 04 10:29:06.653294 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.653034 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 04 10:29:06.653655 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.653631 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 04 10:29:06.653655 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.653633 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 04 10:29:06.661510 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.661485 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 04 10:29:06.685764 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.685741 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9"] Aug 04 10:29:06.719436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-telemeter-client-tls\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.719527 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719444 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-metrics-client-ca\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.719527 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqz5\" (UniqueName: \"kubernetes.io/projected/732c78c5-42d8-4d31-8d08-e7bad8a541e6-kube-api-access-sbqz5\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.719634 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-federate-client-tls\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.719706 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-secret-telemeter-client\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.719760 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.719812 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-serving-certs-ca-bundle\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.719812 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.719789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.820940 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.820864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-telemeter-client-tls\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.820940 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.820899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-metrics-client-ca\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.820940 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.820923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqz5\" (UniqueName: \"kubernetes.io/projected/732c78c5-42d8-4d31-8d08-e7bad8a541e6-kube-api-access-sbqz5\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.821178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.820951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-federate-client-tls\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.821178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.821013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-secret-telemeter-client\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.821178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.821042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.821178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.821070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-serving-certs-ca-bundle\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.821178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.821100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.821758 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.821708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-metrics-client-ca\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.822097 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.822018 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.822218 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.822133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/732c78c5-42d8-4d31-8d08-e7bad8a541e6-serving-certs-ca-bundle\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.823919 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.823874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-telemeter-client-tls\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.824095 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.824070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-federate-client-tls\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.824177 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.824104 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.824282 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.824263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/732c78c5-42d8-4d31-8d08-e7bad8a541e6-secret-telemeter-client\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.832586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.832508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqz5\" (UniqueName: \"kubernetes.io/projected/732c78c5-42d8-4d31-8d08-e7bad8a541e6-kube-api-access-sbqz5\") pod \"telemeter-client-6dc7bb5567-pr5k9\" (UID: \"732c78c5-42d8-4d31-8d08-e7bad8a541e6\") " pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:06.963966 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:06.963944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" Aug 04 10:29:07.084226 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:07.084197 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" event={"ID":"bb5ce1e9-daf7-488c-aca8-23c3a12bde78","Type":"ContainerStarted","Data":"db1e082412cc62f482d63f8058b1c090c2f57509f5edb721760ce71d4b919b8b"} Aug 04 10:29:07.084346 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:07.084267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" event={"ID":"bb5ce1e9-daf7-488c-aca8-23c3a12bde78","Type":"ContainerStarted","Data":"b68d8de684c262da15bf856f39e9815208e6d6b3a960502863bfaa6c27e958ec"} Aug 04 10:29:07.084346 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:07.084310 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" event={"ID":"bb5ce1e9-daf7-488c-aca8-23c3a12bde78","Type":"ContainerStarted","Data":"b1b20ba8f80446ff8fba1e06b4c57bdef19539be3d86c0d3c65c779854cb9752"} Aug 04 10:29:07.085298 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:07.085266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" event={"ID":"f225b1e8-2972-4b59-b10a-4d684718e516","Type":"ContainerStarted","Data":"b86fcf2b2bf77611e1a84b04fa2a23510e8f385971fa5bf07b510964fddea07a"} Aug 04 10:29:07.119675 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:07.119651 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9"] Aug 04 10:29:07.122505 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:29:07.122483 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732c78c5_42d8_4d31_8d08_e7bad8a541e6.slice/crio-41a68562521bfeacba358220fb707fd82492a3b946ca5f82d591b0375868ab9d WatchSource:0}: Error finding container 41a68562521bfeacba358220fb707fd82492a3b946ca5f82d591b0375868ab9d: Status 404 returned error can't find the container with id 41a68562521bfeacba358220fb707fd82492a3b946ca5f82d591b0375868ab9d Aug 04 10:29:08.090543 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:08.090512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" event={"ID":"732c78c5-42d8-4d31-8d08-e7bad8a541e6","Type":"ContainerStarted","Data":"41a68562521bfeacba358220fb707fd82492a3b946ca5f82d591b0375868ab9d"} Aug 04 10:29:08.116713 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:08.116669 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" podStartSLOduration=2.377698429 podStartE2EDuration="5.116652475s" totalImagesPullingTime="2.738954046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:29:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:08.115907893 +0000 UTC m=+109.067315442" watchObservedRunningTime="2026-08-04 10:29:08.116652475 +0000 UTC m=+109.068060024" Aug 04 10:29:09.097734 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:09.097648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" event={"ID":"f225b1e8-2972-4b59-b10a-4d684718e516","Type":"ContainerStarted","Data":"d7706d97f46e00580e3f76b2643d007b2b5d907c4948416048da34072c41d9ba"} Aug 04 10:29:09.098170 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:09.097741 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:09.099196 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:09.099168 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" event={"ID":"732c78c5-42d8-4d31-8d08-e7bad8a541e6","Type":"ContainerStarted","Data":"4b917a49ba882e8caf7cbeb1d1be30fa328a4ac4911e2dbe471162bbeab5d4c6"} Aug 04 10:29:09.099591 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:09.099576 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:09.102953 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:09.102932 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" Aug 04 10:29:09.122458 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:09.122420 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-h2hf5" podStartSLOduration=1.660320462 podStartE2EDuration="3.122405679s" totalImagesPullingTime="1.462085217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:09.121380682 +0000 UTC m=+110.072788229" watchObservedRunningTime="2026-08-04 10:29:09.122405679 +0000 UTC m=+110.073813267" Aug 04 10:29:10.044776 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:10.044751 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h8frs" Aug 04 10:29:10.104709 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:10.104636 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" event={"ID":"732c78c5-42d8-4d31-8d08-e7bad8a541e6","Type":"ContainerStarted","Data":"d1b8da251c562992667a683edde3ceceec45406a85189379a14d247b9b5be49c"} Aug 04 10:29:10.104709 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:10.104688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" event={"ID":"732c78c5-42d8-4d31-8d08-e7bad8a541e6","Type":"ContainerStarted","Data":"129e235cec61f33742595c5eef67c9cac929b0b7d52137353ba5c23cabc311ae"} Aug 04 10:29:10.111861 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:10.111825 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bd5675968-grpgj" Aug 04 10:29:10.131334 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:10.131295 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dc7bb5567-pr5k9" podStartSLOduration=1.485261649 podStartE2EDuration="4.131280749s" totalImagesPullingTime="2.6460191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:29:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:10.129743588 +0000 UTC m=+111.081151137" watchObservedRunningTime="2026-08-04 10:29:10.131280749 +0000 UTC m=+111.082688297" Aug 04 10:29:10.834197 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:10.834161 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5979db58b7-mr9p5"] Aug 04 10:29:12.710564 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.710530 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c5664dc45-6vpcn"] Aug 04 10:29:12.713898 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.713883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.716237 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716218 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 10:29:12.716352 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716219 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 10:29:12.716794 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716777 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 10:29:12.716899 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716788 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 10:29:12.716899 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716788 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r84x5\"" Aug 04 10:29:12.716899 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716869 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 10:29:12.716899 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716855 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 10:29:12.717089 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.716856 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 10:29:12.724270 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.724252 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5664dc45-6vpcn"] Aug 04 10:29:12.772024 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.772003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-serving-cert\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.772105 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.772027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5pv\" (UniqueName: \"kubernetes.io/projected/8cfcc3e1-f504-4d38-aba1-b055de8de279-kube-api-access-gd5pv\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.772105 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.772055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-service-ca\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.772165 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.772150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-oauth-config\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.772215 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.772201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-config\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.772249 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.772227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-oauth-serving-cert\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.873440 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.873417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-config\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.873515 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.873443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-oauth-serving-cert\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.873515 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.873466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-serving-cert\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.873515 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.873479 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5pv\" (UniqueName: \"kubernetes.io/projected/8cfcc3e1-f504-4d38-aba1-b055de8de279-kube-api-access-gd5pv\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.873515 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.873506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-service-ca\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.873652 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.873557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-oauth-config\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.874231 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.874213 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-service-ca\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.874317 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.874231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-oauth-serving-cert\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.874317 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.874231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-config\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.875879 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.875858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-oauth-config\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.875949 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.875882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-serving-cert\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:12.884414 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:12.884390 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5pv\" (UniqueName: \"kubernetes.io/projected/8cfcc3e1-f504-4d38-aba1-b055de8de279-kube-api-access-gd5pv\") pod \"console-c5664dc45-6vpcn\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:13.022189 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:13.022173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:13.151391 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:13.151355 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5664dc45-6vpcn"] Aug 04 10:29:13.154377 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:29:13.154350 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfcc3e1_f504_4d38_aba1_b055de8de279.slice/crio-9ef8631b072d7eeebecaa7cffc2e9bdbd9e999d8dcc3ddb347ea60ce6283c719 WatchSource:0}: Error finding container 9ef8631b072d7eeebecaa7cffc2e9bdbd9e999d8dcc3ddb347ea60ce6283c719: Status 404 returned error can't find the container with id 9ef8631b072d7eeebecaa7cffc2e9bdbd9e999d8dcc3ddb347ea60ce6283c719 Aug 04 10:29:14.125261 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:14.125223 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5664dc45-6vpcn" event={"ID":"8cfcc3e1-f504-4d38-aba1-b055de8de279","Type":"ContainerStarted","Data":"9ef8631b072d7eeebecaa7cffc2e9bdbd9e999d8dcc3ddb347ea60ce6283c719"} Aug 04 10:29:16.138387 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:16.138352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5664dc45-6vpcn" event={"ID":"8cfcc3e1-f504-4d38-aba1-b055de8de279","Type":"ContainerStarted","Data":"7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198"} Aug 04 10:29:16.155961 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:16.155905 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c5664dc45-6vpcn" podStartSLOduration=1.279231697 podStartE2EDuration="4.155884084s" totalImagesPullingTime="2.876652387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:16.155456477 +0000 UTC m=+117.106864025" watchObservedRunningTime="2026-08-04 10:29:16.155884084 +0000 UTC m=+117.107291633" Aug 04 10:29:20.839888 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:20.839857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:29:23.023272 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:23.023239 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:23.023706 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:23.023323 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:23.028145 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:23.028123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:23.167642 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:23.167618 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:29.308452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:29.308423 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:29:29.310617 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:29.310597 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fa238f6-a750-4b82-8c0c-d31e0fcda574-metrics-certs\") pod \"network-metrics-daemon-xrhhd\" (UID: \"0fa238f6-a750-4b82-8c0c-d31e0fcda574\") " pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:29:29.347023 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:29.346999 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p48mn\"" Aug 04 10:29:29.354708 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:29.354693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xrhhd" Aug 04 10:29:29.479014 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:29.478980 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xrhhd"] Aug 04 10:29:29.483724 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:29:29.483697 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa238f6_a750_4b82_8c0c_d31e0fcda574.slice/crio-b1769e1bd787cd5cf489e0d3009b057930e416613418e72780c48c87e8b5ec3e WatchSource:0}: Error finding container b1769e1bd787cd5cf489e0d3009b057930e416613418e72780c48c87e8b5ec3e: Status 404 returned error can't find the container with id b1769e1bd787cd5cf489e0d3009b057930e416613418e72780c48c87e8b5ec3e Aug 04 10:29:30.194590 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:30.194551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrhhd" event={"ID":"0fa238f6-a750-4b82-8c0c-d31e0fcda574","Type":"ContainerStarted","Data":"b1769e1bd787cd5cf489e0d3009b057930e416613418e72780c48c87e8b5ec3e"} Aug 04 10:29:31.200234 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:31.200199 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrhhd" event={"ID":"0fa238f6-a750-4b82-8c0c-d31e0fcda574","Type":"ContainerStarted","Data":"392cbdfc4810e5a57fb95a1d342affccb296a7bbeb1313c939c0f4a643576116"} Aug 04 10:29:31.200595 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:31.200259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xrhhd" event={"ID":"0fa238f6-a750-4b82-8c0c-d31e0fcda574","Type":"ContainerStarted","Data":"0de42a3c779b5206481f2df2847f6ad464e2bf1451b8380d810f4c3d3098bf04"} Aug 04 10:29:31.227368 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:31.227323 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xrhhd" podStartSLOduration=131.331724912 podStartE2EDuration="2m12.227305168s" totalImagesPullingTime="895.580256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:29:31.225550949 +0000 UTC m=+132.176958500" watchObservedRunningTime="2026-08-04 10:29:31.227305168 +0000 UTC m=+132.178712716" Aug 04 10:29:31.638432 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:31.638401 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5664dc45-6vpcn"] Aug 04 10:29:35.852378 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:35.852339 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" podUID="777e50d7-624b-4a82-99c5-56675f43183a" containerName="registry" containerID="cri-o://8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf" gracePeriod=30 Aug 04 10:29:36.090294 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.090273 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:29:36.162463 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162386 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-bound-sa-token\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.162463 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162419 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.162653 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162464 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-installation-pull-secrets\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.162653 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wgln\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-kube-api-access-4wgln\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.162653 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162602 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-registry-certificates\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.162653 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162634 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-image-registry-private-configuration\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.162870 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162700 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/777e50d7-624b-4a82-99c5-56675f43183a-ca-trust-extracted\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.162870 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.162775 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-trusted-ca\") pod \"777e50d7-624b-4a82-99c5-56675f43183a\" (UID: \"777e50d7-624b-4a82-99c5-56675f43183a\") " Aug 04 10:29:36.163162 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.163044 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:29:36.163898 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.163441 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:29:36.165261 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.165233 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:29:36.165424 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.165296 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:29:36.165424 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.165328 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:29:36.165605 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.165584 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-kube-api-access-4wgln" (OuterVolumeSpecName: "kube-api-access-4wgln") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "kube-api-access-4wgln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:29:36.165754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.165731 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:29:36.173409 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.173383 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777e50d7-624b-4a82-99c5-56675f43183a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "777e50d7-624b-4a82-99c5-56675f43183a" (UID: "777e50d7-624b-4a82-99c5-56675f43183a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:29:36.222124 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.222095 2577 generic.go:356] "Generic (PLEG): container finished" podID="777e50d7-624b-4a82-99c5-56675f43183a" containerID="8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf" exitCode=0 Aug 04 10:29:36.222218 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.222138 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" Aug 04 10:29:36.222218 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.222180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" event={"ID":"777e50d7-624b-4a82-99c5-56675f43183a","Type":"ContainerDied","Data":"8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf"} Aug 04 10:29:36.222305 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.222224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5979db58b7-mr9p5" event={"ID":"777e50d7-624b-4a82-99c5-56675f43183a","Type":"ContainerDied","Data":"3af84a67a326a70a5227d68f57e1c70aa17b8dd2e424f37aae6bbe4657188fa2"} Aug 04 10:29:36.222305 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.222244 2577 scope.go:117] "RemoveContainer" containerID="8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf" Aug 04 10:29:36.223744 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.223724 2577 generic.go:356] "Generic (PLEG): container finished" podID="94b88d24-64e8-4d0e-98b4-96c9f17a2d85" containerID="ac0d320aac81aacb441e5193c4d37599c0b77aa86f96838548d41481fdf6fcf2" exitCode=0 Aug 04 10:29:36.223854 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.223807 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" event={"ID":"94b88d24-64e8-4d0e-98b4-96c9f17a2d85","Type":"ContainerDied","Data":"ac0d320aac81aacb441e5193c4d37599c0b77aa86f96838548d41481fdf6fcf2"} Aug 04 10:29:36.224180 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.224159 2577 scope.go:117] "RemoveContainer" containerID="ac0d320aac81aacb441e5193c4d37599c0b77aa86f96838548d41481fdf6fcf2" Aug 04 10:29:36.225481 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.225458 2577 generic.go:356] "Generic (PLEG): container finished" podID="e94db208-1619-40c0-9b02-9bfb74cb474f" containerID="11b47aaaa8ccd522383cf272f38b37cbf13786f6897bfc918836a897efb3d351" exitCode=0 Aug 04 10:29:36.225574 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.225509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-ws6m5" event={"ID":"e94db208-1619-40c0-9b02-9bfb74cb474f","Type":"ContainerDied","Data":"11b47aaaa8ccd522383cf272f38b37cbf13786f6897bfc918836a897efb3d351"} Aug 04 10:29:36.225880 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.225864 2577 scope.go:117] "RemoveContainer" containerID="11b47aaaa8ccd522383cf272f38b37cbf13786f6897bfc918836a897efb3d351" Aug 04 10:29:36.263084 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263052 2577 scope.go:117] "RemoveContainer" containerID="8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf" Aug 04 10:29:36.263400 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:29:36.263373 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf\": container with ID starting with 8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf not found: ID does not exist" containerID="8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf" Aug 04 10:29:36.263475 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263412 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf"} err="failed to get container status \"8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf\": rpc error: code = NotFound desc = could not find container \"8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf\": container with ID starting with 8675cdbb0ddbdb73b4819c946718528974404da7a3cdcbfb9ddeb6499fca4ecf not found: ID does not exist" Aug 04 10:29:36.263475 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263382 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-trusted-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.263475 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263456 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-bound-sa-token\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.263475 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263466 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-registry-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.263475 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263474 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-installation-pull-secrets\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.263626 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263483 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wgln\" (UniqueName: \"kubernetes.io/projected/777e50d7-624b-4a82-99c5-56675f43183a-kube-api-access-4wgln\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.263626 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263492 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/777e50d7-624b-4a82-99c5-56675f43183a-registry-certificates\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.263626 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263500 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/777e50d7-624b-4a82-99c5-56675f43183a-image-registry-private-configuration\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.263626 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.263522 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/777e50d7-624b-4a82-99c5-56675f43183a-ca-trust-extracted\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:36.298427 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.298402 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5979db58b7-mr9p5"] Aug 04 10:29:36.301774 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:36.301754 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5979db58b7-mr9p5"] Aug 04 10:29:37.192220 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:37.192143 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-587b49775d-8b8rl_23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a/router/0.log" Aug 04 10:29:37.204539 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:37.204500 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rfcm6_16a59ee9-ea69-403e-8ee7-06492cb61c78/serve-healthcheck-canary/0.log" Aug 04 10:29:37.238279 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:37.238247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-4grhn" event={"ID":"94b88d24-64e8-4d0e-98b4-96c9f17a2d85","Type":"ContainerStarted","Data":"a197a58f241212a12436d15ca825ef7c1b39d966bf86bee578926c31d984cf9f"} Aug 04 10:29:37.240312 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:37.240285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-ws6m5" event={"ID":"e94db208-1619-40c0-9b02-9bfb74cb474f","Type":"ContainerStarted","Data":"233b7c594c65b44808c245d858838ec77ea7c2e12a8d7d153477442e4fbed068"} Aug 04 10:29:37.531035 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:37.530998 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777e50d7-624b-4a82-99c5-56675f43183a" path="/var/lib/kubelet/pods/777e50d7-624b-4a82-99c5-56675f43183a/volumes" Aug 04 10:29:47.285274 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:47.285241 2577 generic.go:356] "Generic (PLEG): container finished" podID="779fd46d-da59-4807-9385-607c2dfda848" containerID="5235c980aea41d3da6406ea5f5aa525e05df8d6d977e09323f7a1437e88e2244" exitCode=0 Aug 04 10:29:47.285748 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:47.285320 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" event={"ID":"779fd46d-da59-4807-9385-607c2dfda848","Type":"ContainerDied","Data":"5235c980aea41d3da6406ea5f5aa525e05df8d6d977e09323f7a1437e88e2244"} Aug 04 10:29:47.285748 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:47.285734 2577 scope.go:117] "RemoveContainer" containerID="5235c980aea41d3da6406ea5f5aa525e05df8d6d977e09323f7a1437e88e2244" Aug 04 10:29:48.291147 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:48.291114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mqssx" event={"ID":"779fd46d-da59-4807-9385-607c2dfda848","Type":"ContainerStarted","Data":"a60e84adee599cc90439194f4b7547cce106dc2434ec9d65863b24b304dd93c8"} Aug 04 10:29:56.657765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:56.657703 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c5664dc45-6vpcn" podUID="8cfcc3e1-f504-4d38-aba1-b055de8de279" containerName="console" containerID="cri-o://7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198" gracePeriod=15 Aug 04 10:29:56.897864 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:56.897826 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:57.024599 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.024574 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-oauth-serving-cert\") pod \"8cfcc3e1-f504-4d38-aba1-b055de8de279\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " Aug 04 10:29:57.024754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.024615 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-serving-cert\") pod \"8cfcc3e1-f504-4d38-aba1-b055de8de279\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " Aug 04 10:29:57.024754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.024652 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd5pv\" (UniqueName: \"kubernetes.io/projected/8cfcc3e1-f504-4d38-aba1-b055de8de279-kube-api-access-gd5pv\") pod \"8cfcc3e1-f504-4d38-aba1-b055de8de279\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " Aug 04 10:29:57.024754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.024680 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-oauth-config\") pod \"8cfcc3e1-f504-4d38-aba1-b055de8de279\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " Aug 04 10:29:57.024754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.024714 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-service-ca\") pod \"8cfcc3e1-f504-4d38-aba1-b055de8de279\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " Aug 04 10:29:57.024754 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.024744 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-config\") pod \"8cfcc3e1-f504-4d38-aba1-b055de8de279\" (UID: \"8cfcc3e1-f504-4d38-aba1-b055de8de279\") " Aug 04 10:29:57.025149 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.025112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8cfcc3e1-f504-4d38-aba1-b055de8de279" (UID: "8cfcc3e1-f504-4d38-aba1-b055de8de279"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:29:57.025272 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.025116 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-service-ca" (OuterVolumeSpecName: "service-ca") pod "8cfcc3e1-f504-4d38-aba1-b055de8de279" (UID: "8cfcc3e1-f504-4d38-aba1-b055de8de279"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:29:57.025272 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.025207 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-config" (OuterVolumeSpecName: "console-config") pod "8cfcc3e1-f504-4d38-aba1-b055de8de279" (UID: "8cfcc3e1-f504-4d38-aba1-b055de8de279"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:29:57.026976 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.026948 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8cfcc3e1-f504-4d38-aba1-b055de8de279" (UID: "8cfcc3e1-f504-4d38-aba1-b055de8de279"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:29:57.027062 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.027001 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfcc3e1-f504-4d38-aba1-b055de8de279-kube-api-access-gd5pv" (OuterVolumeSpecName: "kube-api-access-gd5pv") pod "8cfcc3e1-f504-4d38-aba1-b055de8de279" (UID: "8cfcc3e1-f504-4d38-aba1-b055de8de279"). InnerVolumeSpecName "kube-api-access-gd5pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:29:57.027062 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.027024 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8cfcc3e1-f504-4d38-aba1-b055de8de279" (UID: "8cfcc3e1-f504-4d38-aba1-b055de8de279"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:29:57.125409 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.125381 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-oauth-config\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:57.125409 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.125402 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:57.125409 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.125411 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-config\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:57.125604 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.125419 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8cfcc3e1-f504-4d38-aba1-b055de8de279-oauth-serving-cert\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:57.125604 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.125427 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8cfcc3e1-f504-4d38-aba1-b055de8de279-console-serving-cert\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:57.125604 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.125435 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd5pv\" (UniqueName: \"kubernetes.io/projected/8cfcc3e1-f504-4d38-aba1-b055de8de279-kube-api-access-gd5pv\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:29:57.330731 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.330655 2577 generic.go:356] "Generic (PLEG): container finished" podID="8cfcc3e1-f504-4d38-aba1-b055de8de279" containerID="7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198" exitCode=0 Aug 04 10:29:57.330731 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.330701 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5664dc45-6vpcn" Aug 04 10:29:57.330902 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.330739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5664dc45-6vpcn" event={"ID":"8cfcc3e1-f504-4d38-aba1-b055de8de279","Type":"ContainerDied","Data":"7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198"} Aug 04 10:29:57.330902 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.330794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5664dc45-6vpcn" event={"ID":"8cfcc3e1-f504-4d38-aba1-b055de8de279","Type":"ContainerDied","Data":"9ef8631b072d7eeebecaa7cffc2e9bdbd9e999d8dcc3ddb347ea60ce6283c719"} Aug 04 10:29:57.330902 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.330821 2577 scope.go:117] "RemoveContainer" containerID="7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198" Aug 04 10:29:57.340120 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.340103 2577 scope.go:117] "RemoveContainer" containerID="7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198" Aug 04 10:29:57.340367 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:29:57.340348 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198\": container with ID starting with 7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198 not found: ID does not exist" containerID="7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198" Aug 04 10:29:57.340411 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.340376 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198"} err="failed to get container status \"7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198\": rpc error: code = NotFound desc = could not find container \"7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198\": container with ID starting with 7a26db2559e1c8dba048d14a1bf864bfdc51cdb709957f4d9b6a0de0a2a15198 not found: ID does not exist" Aug 04 10:29:57.351738 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.351715 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5664dc45-6vpcn"] Aug 04 10:29:57.356029 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.356013 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c5664dc45-6vpcn"] Aug 04 10:29:57.532953 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:29:57.532925 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cfcc3e1-f504-4d38-aba1-b055de8de279" path="/var/lib/kubelet/pods/8cfcc3e1-f504-4d38-aba1-b055de8de279/volumes" Aug 04 10:30:23.582658 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.582625 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:30:23.583073 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.582936 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777e50d7-624b-4a82-99c5-56675f43183a" containerName="registry" Aug 04 10:30:23.583073 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.582948 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="777e50d7-624b-4a82-99c5-56675f43183a" containerName="registry" Aug 04 10:30:23.583073 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.582964 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfcc3e1-f504-4d38-aba1-b055de8de279" containerName="console" Aug 04 10:30:23.583073 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.582970 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfcc3e1-f504-4d38-aba1-b055de8de279" containerName="console" Aug 04 10:30:23.583073 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.583021 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="777e50d7-624b-4a82-99c5-56675f43183a" containerName="registry" Aug 04 10:30:23.583073 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.583030 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cfcc3e1-f504-4d38-aba1-b055de8de279" containerName="console" Aug 04 10:30:23.587780 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.587764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.589733 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.589701 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 10:30:23.589866 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.589737 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xx78f\"" Aug 04 10:30:23.589866 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.589710 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 10:30:23.589989 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.589879 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 10:30:23.590075 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.590053 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 10:30:23.590378 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.590363 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 10:30:23.590481 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.590453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 10:30:23.590564 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.590547 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 10:30:23.591054 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.591035 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 10:30:23.596123 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.596106 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 10:30:23.602081 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.602063 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:30:23.623077 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-web-config\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623170 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623170 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623242 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623242 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-config-out\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623303 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623303 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623303 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623389 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9c8r\" (UniqueName: \"kubernetes.io/projected/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-kube-api-access-f9c8r\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623389 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623389 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623389 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.623502 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.623412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724100 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-web-config\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724100 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724296 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724296 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724296 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-config-out\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724444 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724444 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724444 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724444 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9c8r\" (UniqueName: \"kubernetes.io/projected/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-kube-api-access-f9c8r\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724624 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724624 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724624 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724624 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.724624 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.724553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.725313 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.725289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.726520 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.726496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.727513 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.727130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-config-out\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.727513 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.727461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-web-config\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.727720 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.727521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.727720 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.727523 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.727943 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.727912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.728061 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.728045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.728247 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.728226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.728875 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.728853 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.728947 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.728921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.733576 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.733554 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9c8r\" (UniqueName: \"kubernetes.io/projected/f82a49fc-fcb3-4c3f-ba96-f96b403b40a4-kube-api-access-f9c8r\") pod \"alertmanager-main-0\" (UID: \"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:23.897901 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:23.897828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 10:30:24.031353 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:24.031318 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 10:30:24.033596 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:30:24.033565 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82a49fc_fcb3_4c3f_ba96_f96b403b40a4.slice/crio-af51c63ea67c0d1eb2d5858db0020df789eae0feb4b21005b91913236ac0c9ab WatchSource:0}: Error finding container af51c63ea67c0d1eb2d5858db0020df789eae0feb4b21005b91913236ac0c9ab: Status 404 returned error can't find the container with id af51c63ea67c0d1eb2d5858db0020df789eae0feb4b21005b91913236ac0c9ab Aug 04 10:30:24.438530 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:24.438494 2577 generic.go:356] "Generic (PLEG): container finished" podID="f82a49fc-fcb3-4c3f-ba96-f96b403b40a4" containerID="7754a69a6cde93a09bc191308aabcc33c9dc11f1895e64036160dd68f9d46499" exitCode=0 Aug 04 10:30:24.438688 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:24.438567 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerDied","Data":"7754a69a6cde93a09bc191308aabcc33c9dc11f1895e64036160dd68f9d46499"} Aug 04 10:30:24.438688 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:24.438664 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerStarted","Data":"af51c63ea67c0d1eb2d5858db0020df789eae0feb4b21005b91913236ac0c9ab"} Aug 04 10:30:26.450913 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:26.450880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerStarted","Data":"ff9a3ba9c3f5c0c49e7bc3cc669b89eecf5ff58613376344631602fe9b7b4fd8"} Aug 04 10:30:26.451262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:26.450947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerStarted","Data":"e1bdb673ce9cd4a72b78daea27196b80273c738effd23c9bd3aa90bc4cbf2df5"} Aug 04 10:30:26.451262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:26.450971 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerStarted","Data":"1fde419c26317d4ceb4e980cca6f5c9bc4a9112e3c9f895380d09442d32c57a2"} Aug 04 10:30:26.451262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:26.450994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerStarted","Data":"38f93f382ce5c9ab37ddadd72af1ee08785bb32ed6abb70a4f3e0c42c5b25afd"} Aug 04 10:30:26.451262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:26.451017 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerStarted","Data":"28b9373f39a21de9cdc37c4053e54a3db9f56bf9a7bea45e667bbf979f2bc435"} Aug 04 10:30:26.451262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:26.451039 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f82a49fc-fcb3-4c3f-ba96-f96b403b40a4","Type":"ContainerStarted","Data":"edeecf4467e899a01dec6a55165360ab095486974f07d49091c687fc557f4eba"} Aug 04 10:30:26.487989 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:26.487934 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.871159065 podStartE2EDuration="3.487912726s" totalImagesPullingTime="1.600550846s" totalInitContainerRuntime="16.202815ms" isStatefulPod=false podCreationTimestamp="2026-08-04 10:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:30:26.486250933 +0000 UTC m=+187.437658482" watchObservedRunningTime="2026-08-04 10:30:26.487912726 +0000 UTC m=+187.439320276" Aug 04 10:30:54.535639 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.535605 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75cdf679f5-7vhpm"] Aug 04 10:30:54.539076 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.539051 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.541641 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.541622 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 10:30:54.542050 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.542033 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 10:30:54.542277 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.542048 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 10:30:54.542438 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.542139 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 10:30:54.542438 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.542180 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 10:30:54.543102 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.542926 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 10:30:54.543102 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.542936 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r84x5\"" Aug 04 10:30:54.543253 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.543201 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 10:30:54.548883 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.548864 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 10:30:54.551498 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.551477 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75cdf679f5-7vhpm"] Aug 04 10:30:54.663027 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.662998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-trusted-ca-bundle\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.663130 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.663045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-oauth-config\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.663130 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.663070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-service-ca\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.663130 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.663094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-oauth-serving-cert\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.663130 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.663124 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-config\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.663265 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.663145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s4l7\" (UniqueName: \"kubernetes.io/projected/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-kube-api-access-4s4l7\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.663265 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.663180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-serving-cert\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764012 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.763986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-serving-cert\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764114 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-trusted-ca-bundle\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764114 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-oauth-config\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764114 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-service-ca\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764114 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-oauth-serving-cert\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764299 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-config\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764299 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s4l7\" (UniqueName: \"kubernetes.io/projected/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-kube-api-access-4s4l7\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764792 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764770 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-service-ca\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.764964 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-oauth-serving-cert\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.765020 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.764998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-config\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.765071 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.765058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-trusted-ca-bundle\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.766445 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.766415 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-oauth-config\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.766647 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.766626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-serving-cert\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.772452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.772436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s4l7\" (UniqueName: \"kubernetes.io/projected/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-kube-api-access-4s4l7\") pod \"console-75cdf679f5-7vhpm\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.851179 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.851127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:30:54.979367 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:54.979341 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75cdf679f5-7vhpm"] Aug 04 10:30:54.981798 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:30:54.981776 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aac9f2_7c28_4e59_9b8f_63792b08ca49.slice/crio-84dda9b3b5f496f20674638c02d3bf1cedf2c0de9a998897afe0e29c3d8468b1 WatchSource:0}: Error finding container 84dda9b3b5f496f20674638c02d3bf1cedf2c0de9a998897afe0e29c3d8468b1: Status 404 returned error can't find the container with id 84dda9b3b5f496f20674638c02d3bf1cedf2c0de9a998897afe0e29c3d8468b1 Aug 04 10:30:55.568023 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:55.567987 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cdf679f5-7vhpm" event={"ID":"b3aac9f2-7c28-4e59-9b8f-63792b08ca49","Type":"ContainerStarted","Data":"11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869"} Aug 04 10:30:55.568381 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:55.568043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cdf679f5-7vhpm" event={"ID":"b3aac9f2-7c28-4e59-9b8f-63792b08ca49","Type":"ContainerStarted","Data":"84dda9b3b5f496f20674638c02d3bf1cedf2c0de9a998897afe0e29c3d8468b1"} Aug 04 10:30:55.586637 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:30:55.586590 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75cdf679f5-7vhpm" podStartSLOduration=1.586575883 podStartE2EDuration="1.586575883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:30:55.585643416 +0000 UTC m=+216.537050964" watchObservedRunningTime="2026-08-04 10:30:55.586575883 +0000 UTC m=+216.537983431" Aug 04 10:31:04.851377 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:04.851344 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:31:04.851865 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:04.851634 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:31:04.856136 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:04.856116 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:31:05.609527 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:05.609497 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:31:21.755811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.755775 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45"] Aug 04 10:31:21.759410 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.759392 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.761788 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.761766 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:31:21.761929 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.761823 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:31:21.762636 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.762618 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6dcj4\"" Aug 04 10:31:21.767719 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.767697 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45"] Aug 04 10:31:21.785007 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.784981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.785116 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.785012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvk6v\" (UniqueName: \"kubernetes.io/projected/12fb7807-837c-42dc-a545-ba71964b0859-kube-api-access-nvk6v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.785116 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.785094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.885687 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.885648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.885882 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.885720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.885882 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.885751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvk6v\" (UniqueName: \"kubernetes.io/projected/12fb7807-837c-42dc-a545-ba71964b0859-kube-api-access-nvk6v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.886153 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.886131 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.886192 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.886145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:21.895312 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:21.895282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvk6v\" (UniqueName: \"kubernetes.io/projected/12fb7807-837c-42dc-a545-ba71964b0859-kube-api-access-nvk6v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:22.069913 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:22.069816 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:22.201049 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:22.201016 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45"] Aug 04 10:31:22.204045 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:31:22.204018 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fb7807_837c_42dc_a545_ba71964b0859.slice/crio-02778ad04aed96be7814237878c527f370db5fef97d2787e10f95bbf64421421 WatchSource:0}: Error finding container 02778ad04aed96be7814237878c527f370db5fef97d2787e10f95bbf64421421: Status 404 returned error can't find the container with id 02778ad04aed96be7814237878c527f370db5fef97d2787e10f95bbf64421421 Aug 04 10:31:22.672985 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:22.672952 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" event={"ID":"12fb7807-837c-42dc-a545-ba71964b0859","Type":"ContainerStarted","Data":"02778ad04aed96be7814237878c527f370db5fef97d2787e10f95bbf64421421"} Aug 04 10:31:28.703455 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:28.703415 2577 generic.go:356] "Generic (PLEG): container finished" podID="12fb7807-837c-42dc-a545-ba71964b0859" containerID="3cc0eb5e5cf6c7bf1ed47ed3452812d7c8e37446ebf33d3f457a0444eacb257b" exitCode=0 Aug 04 10:31:28.703885 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:28.703498 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" event={"ID":"12fb7807-837c-42dc-a545-ba71964b0859","Type":"ContainerDied","Data":"3cc0eb5e5cf6c7bf1ed47ed3452812d7c8e37446ebf33d3f457a0444eacb257b"} Aug 04 10:31:32.722396 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:32.722358 2577 generic.go:356] "Generic (PLEG): container finished" podID="12fb7807-837c-42dc-a545-ba71964b0859" containerID="9cce9eb9b8855399220979f0962158d4bfb2d475e377ab89e0d7b60567434f8e" exitCode=0 Aug 04 10:31:32.722855 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:32.722443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" event={"ID":"12fb7807-837c-42dc-a545-ba71964b0859","Type":"ContainerDied","Data":"9cce9eb9b8855399220979f0962158d4bfb2d475e377ab89e0d7b60567434f8e"} Aug 04 10:31:39.759115 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:39.759080 2577 generic.go:356] "Generic (PLEG): container finished" podID="12fb7807-837c-42dc-a545-ba71964b0859" containerID="c9a57a9f29a74dee8710d41b0363e88a5d226992e8d06ffee1b0cdaec8e13b4b" exitCode=0 Aug 04 10:31:39.759535 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:39.759131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" event={"ID":"12fb7807-837c-42dc-a545-ba71964b0859","Type":"ContainerDied","Data":"c9a57a9f29a74dee8710d41b0363e88a5d226992e8d06ffee1b0cdaec8e13b4b"} Aug 04 10:31:40.890199 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:40.890169 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:40.960778 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:40.960747 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvk6v\" (UniqueName: \"kubernetes.io/projected/12fb7807-837c-42dc-a545-ba71964b0859-kube-api-access-nvk6v\") pod \"12fb7807-837c-42dc-a545-ba71964b0859\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " Aug 04 10:31:40.960975 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:40.960795 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-util\") pod \"12fb7807-837c-42dc-a545-ba71964b0859\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " Aug 04 10:31:40.960975 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:40.960858 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-bundle\") pod \"12fb7807-837c-42dc-a545-ba71964b0859\" (UID: \"12fb7807-837c-42dc-a545-ba71964b0859\") " Aug 04 10:31:40.961517 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:40.961492 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-bundle" (OuterVolumeSpecName: "bundle") pod "12fb7807-837c-42dc-a545-ba71964b0859" (UID: "12fb7807-837c-42dc-a545-ba71964b0859"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:31:40.962978 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:40.962956 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fb7807-837c-42dc-a545-ba71964b0859-kube-api-access-nvk6v" (OuterVolumeSpecName: "kube-api-access-nvk6v") pod "12fb7807-837c-42dc-a545-ba71964b0859" (UID: "12fb7807-837c-42dc-a545-ba71964b0859"). InnerVolumeSpecName "kube-api-access-nvk6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:31:40.965018 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:40.964993 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-util" (OuterVolumeSpecName: "util") pod "12fb7807-837c-42dc-a545-ba71964b0859" (UID: "12fb7807-837c-42dc-a545-ba71964b0859"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:31:41.062305 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:41.062225 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:31:41.062305 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:41.062255 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvk6v\" (UniqueName: \"kubernetes.io/projected/12fb7807-837c-42dc-a545-ba71964b0859-kube-api-access-nvk6v\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:31:41.062305 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:41.062265 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12fb7807-837c-42dc-a545-ba71964b0859-util\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:31:41.769921 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:41.769888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" event={"ID":"12fb7807-837c-42dc-a545-ba71964b0859","Type":"ContainerDied","Data":"02778ad04aed96be7814237878c527f370db5fef97d2787e10f95bbf64421421"} Aug 04 10:31:41.770110 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:41.769927 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754077c45" Aug 04 10:31:41.770110 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:41.769935 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02778ad04aed96be7814237878c527f370db5fef97d2787e10f95bbf64421421" Aug 04 10:31:44.713587 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713507 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d"] Aug 04 10:31:44.714053 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713854 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12fb7807-837c-42dc-a545-ba71964b0859" containerName="pull" Aug 04 10:31:44.714053 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713865 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12fb7807-837c-42dc-a545-ba71964b0859" containerName="pull" Aug 04 10:31:44.714053 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713873 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12fb7807-837c-42dc-a545-ba71964b0859" containerName="extract" Aug 04 10:31:44.714053 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713878 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12fb7807-837c-42dc-a545-ba71964b0859" containerName="extract" Aug 04 10:31:44.714053 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713896 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12fb7807-837c-42dc-a545-ba71964b0859" containerName="util" Aug 04 10:31:44.714053 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713902 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12fb7807-837c-42dc-a545-ba71964b0859" containerName="util" Aug 04 10:31:44.714053 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.713961 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="12fb7807-837c-42dc-a545-ba71964b0859" containerName="extract" Aug 04 10:31:44.761172 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.761137 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d"] Aug 04 10:31:44.761323 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.761234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:44.763427 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.763404 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 10:31:44.763506 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.763404 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:31:44.763506 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.763494 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-96mn4\"" Aug 04 10:31:44.896764 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.896725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzdb\" (UniqueName: \"kubernetes.io/projected/f5e5ee03-7ac2-4b4c-950d-e03617630765-kube-api-access-klzdb\") pod \"cert-manager-operator-controller-manager-55c444bf86-qf47d\" (UID: \"f5e5ee03-7ac2-4b4c-950d-e03617630765\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:44.896980 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.896853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5e5ee03-7ac2-4b4c-950d-e03617630765-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qf47d\" (UID: \"f5e5ee03-7ac2-4b4c-950d-e03617630765\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:44.997474 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.997379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5e5ee03-7ac2-4b4c-950d-e03617630765-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qf47d\" (UID: \"f5e5ee03-7ac2-4b4c-950d-e03617630765\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:44.997474 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.997450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzdb\" (UniqueName: \"kubernetes.io/projected/f5e5ee03-7ac2-4b4c-950d-e03617630765-kube-api-access-klzdb\") pod \"cert-manager-operator-controller-manager-55c444bf86-qf47d\" (UID: \"f5e5ee03-7ac2-4b4c-950d-e03617630765\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:44.997782 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:44.997764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5e5ee03-7ac2-4b4c-950d-e03617630765-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qf47d\" (UID: \"f5e5ee03-7ac2-4b4c-950d-e03617630765\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:45.006857 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:45.006806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzdb\" (UniqueName: \"kubernetes.io/projected/f5e5ee03-7ac2-4b4c-950d-e03617630765-kube-api-access-klzdb\") pod \"cert-manager-operator-controller-manager-55c444bf86-qf47d\" (UID: \"f5e5ee03-7ac2-4b4c-950d-e03617630765\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:45.070786 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:45.070739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" Aug 04 10:31:45.216627 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:45.216591 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d"] Aug 04 10:31:45.220755 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:31:45.220724 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e5ee03_7ac2_4b4c_950d_e03617630765.slice/crio-6526baaef9c4df4f8d66d243948e357e8fe391ef615ec032fe47a349e9a99e2e WatchSource:0}: Error finding container 6526baaef9c4df4f8d66d243948e357e8fe391ef615ec032fe47a349e9a99e2e: Status 404 returned error can't find the container with id 6526baaef9c4df4f8d66d243948e357e8fe391ef615ec032fe47a349e9a99e2e Aug 04 10:31:45.790576 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:45.790538 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" event={"ID":"f5e5ee03-7ac2-4b4c-950d-e03617630765","Type":"ContainerStarted","Data":"6526baaef9c4df4f8d66d243948e357e8fe391ef615ec032fe47a349e9a99e2e"} Aug 04 10:31:47.804670 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:47.804634 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" event={"ID":"f5e5ee03-7ac2-4b4c-950d-e03617630765","Type":"ContainerStarted","Data":"9d6ad1629dcd5ce8dde644bf946ef10ceae74cb06ce492046702e31fef3b09a2"} Aug 04 10:31:47.847967 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:47.847914 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qf47d" podStartSLOduration=2.088101839 podStartE2EDuration="3.847896737s" totalImagesPullingTime="1.759794898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:31:47.84650662 +0000 UTC m=+268.797914170" watchObservedRunningTime="2026-08-04 10:31:47.847896737 +0000 UTC m=+268.799304289" Aug 04 10:31:49.021905 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.021867 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m"] Aug 04 10:31:49.031343 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.031313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.033623 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.033598 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:31:49.034525 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.034478 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6dcj4\"" Aug 04 10:31:49.034868 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.034849 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:31:49.036982 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.036963 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m"] Aug 04 10:31:49.139254 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.139208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.139425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.139266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlffl\" (UniqueName: \"kubernetes.io/projected/94a7727b-b0eb-4e86-a066-6bf5f6acd546-kube-api-access-hlffl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.139425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.139310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.240147 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.240109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.240275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.240152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlffl\" (UniqueName: \"kubernetes.io/projected/94a7727b-b0eb-4e86-a066-6bf5f6acd546-kube-api-access-hlffl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.240275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.240172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.240513 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.240496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.240546 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.240517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.248124 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.248089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlffl\" (UniqueName: \"kubernetes.io/projected/94a7727b-b0eb-4e86-a066-6bf5f6acd546-kube-api-access-hlffl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.341811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.341725 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:49.477325 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.477298 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m"] Aug 04 10:31:49.479783 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:31:49.479751 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a7727b_b0eb_4e86_a066_6bf5f6acd546.slice/crio-45b151de3b857bf2648503d5587ee23828501004b816a4d759eccb836679fddc WatchSource:0}: Error finding container 45b151de3b857bf2648503d5587ee23828501004b816a4d759eccb836679fddc: Status 404 returned error can't find the container with id 45b151de3b857bf2648503d5587ee23828501004b816a4d759eccb836679fddc Aug 04 10:31:49.815170 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.815137 2577 generic.go:356] "Generic (PLEG): container finished" podID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerID="6216b94f312cc5779a62e85fe6edcb4808941d610534e1c3389d24e6a00fc69e" exitCode=0 Aug 04 10:31:49.815356 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.815217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" event={"ID":"94a7727b-b0eb-4e86-a066-6bf5f6acd546","Type":"ContainerDied","Data":"6216b94f312cc5779a62e85fe6edcb4808941d610534e1c3389d24e6a00fc69e"} Aug 04 10:31:49.815356 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:49.815275 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" event={"ID":"94a7727b-b0eb-4e86-a066-6bf5f6acd546","Type":"ContainerStarted","Data":"45b151de3b857bf2648503d5587ee23828501004b816a4d759eccb836679fddc"} Aug 04 10:31:50.746929 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.746756 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s"] Aug 04 10:31:50.750624 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.750595 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:50.753320 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.753119 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 10:31:50.753806 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.753748 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 10:31:50.759764 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.759709 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s"] Aug 04 10:31:50.853301 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.853261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1a275fb-395c-4765-98f9-aecdd23ee618-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2mp6s\" (UID: \"f1a275fb-395c-4765-98f9-aecdd23ee618\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:50.853475 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.853311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8cb\" (UniqueName: \"kubernetes.io/projected/f1a275fb-395c-4765-98f9-aecdd23ee618-kube-api-access-cw8cb\") pod \"cert-manager-cainjector-559cdd8c87-2mp6s\" (UID: \"f1a275fb-395c-4765-98f9-aecdd23ee618\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:50.954652 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.954611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1a275fb-395c-4765-98f9-aecdd23ee618-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2mp6s\" (UID: \"f1a275fb-395c-4765-98f9-aecdd23ee618\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:50.954652 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.954655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8cb\" (UniqueName: \"kubernetes.io/projected/f1a275fb-395c-4765-98f9-aecdd23ee618-kube-api-access-cw8cb\") pod \"cert-manager-cainjector-559cdd8c87-2mp6s\" (UID: \"f1a275fb-395c-4765-98f9-aecdd23ee618\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:50.962556 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.962526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1a275fb-395c-4765-98f9-aecdd23ee618-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2mp6s\" (UID: \"f1a275fb-395c-4765-98f9-aecdd23ee618\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:50.962709 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:50.962688 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8cb\" (UniqueName: \"kubernetes.io/projected/f1a275fb-395c-4765-98f9-aecdd23ee618-kube-api-access-cw8cb\") pod \"cert-manager-cainjector-559cdd8c87-2mp6s\" (UID: \"f1a275fb-395c-4765-98f9-aecdd23ee618\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:51.070582 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:51.070548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" Aug 04 10:31:51.632496 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:51.632421 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s"] Aug 04 10:31:51.634928 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:31:51.634903 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a275fb_395c_4765_98f9_aecdd23ee618.slice/crio-7d07d86869a41b6bbac46dd274758b6a38addee9712366732bdcac857699872e WatchSource:0}: Error finding container 7d07d86869a41b6bbac46dd274758b6a38addee9712366732bdcac857699872e: Status 404 returned error can't find the container with id 7d07d86869a41b6bbac46dd274758b6a38addee9712366732bdcac857699872e Aug 04 10:31:51.824867 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:51.824762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" event={"ID":"f1a275fb-395c-4765-98f9-aecdd23ee618","Type":"ContainerStarted","Data":"7d07d86869a41b6bbac46dd274758b6a38addee9712366732bdcac857699872e"} Aug 04 10:31:51.828951 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:51.828928 2577 generic.go:356] "Generic (PLEG): container finished" podID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerID="44e4179d6c6a24aa00ec942521eaab6a9b9f6430a919952e038aaaed357c066d" exitCode=0 Aug 04 10:31:51.829066 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:51.828986 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" event={"ID":"94a7727b-b0eb-4e86-a066-6bf5f6acd546","Type":"ContainerDied","Data":"44e4179d6c6a24aa00ec942521eaab6a9b9f6430a919952e038aaaed357c066d"} Aug 04 10:31:52.837137 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:52.837086 2577 generic.go:356] "Generic (PLEG): container finished" podID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerID="88aca89157712f67a000ea7c4e5ea42cd8439e7a5d4498be7f5653d140bec66d" exitCode=0 Aug 04 10:31:52.837599 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:52.837163 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" event={"ID":"94a7727b-b0eb-4e86-a066-6bf5f6acd546","Type":"ContainerDied","Data":"88aca89157712f67a000ea7c4e5ea42cd8439e7a5d4498be7f5653d140bec66d"} Aug 04 10:31:54.089008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.088978 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:54.286095 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.286062 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-bundle\") pod \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " Aug 04 10:31:54.286273 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.286154 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-util\") pod \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " Aug 04 10:31:54.286273 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.286194 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlffl\" (UniqueName: \"kubernetes.io/projected/94a7727b-b0eb-4e86-a066-6bf5f6acd546-kube-api-access-hlffl\") pod \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\" (UID: \"94a7727b-b0eb-4e86-a066-6bf5f6acd546\") " Aug 04 10:31:54.286532 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.286504 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-bundle" (OuterVolumeSpecName: "bundle") pod "94a7727b-b0eb-4e86-a066-6bf5f6acd546" (UID: "94a7727b-b0eb-4e86-a066-6bf5f6acd546"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:31:54.288912 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.288882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a7727b-b0eb-4e86-a066-6bf5f6acd546-kube-api-access-hlffl" (OuterVolumeSpecName: "kube-api-access-hlffl") pod "94a7727b-b0eb-4e86-a066-6bf5f6acd546" (UID: "94a7727b-b0eb-4e86-a066-6bf5f6acd546"). InnerVolumeSpecName "kube-api-access-hlffl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:31:54.292043 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.292010 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-util" (OuterVolumeSpecName: "util") pod "94a7727b-b0eb-4e86-a066-6bf5f6acd546" (UID: "94a7727b-b0eb-4e86-a066-6bf5f6acd546"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:31:54.386936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.386812 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:31:54.386936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.386879 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94a7727b-b0eb-4e86-a066-6bf5f6acd546-util\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:31:54.386936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.386894 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlffl\" (UniqueName: \"kubernetes.io/projected/94a7727b-b0eb-4e86-a066-6bf5f6acd546-kube-api-access-hlffl\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:31:54.848472 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.848440 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" Aug 04 10:31:54.848655 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.848471 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxtq4m" event={"ID":"94a7727b-b0eb-4e86-a066-6bf5f6acd546","Type":"ContainerDied","Data":"45b151de3b857bf2648503d5587ee23828501004b816a4d759eccb836679fddc"} Aug 04 10:31:54.848655 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.848526 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45b151de3b857bf2648503d5587ee23828501004b816a4d759eccb836679fddc" Aug 04 10:31:54.850014 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.849988 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" event={"ID":"f1a275fb-395c-4765-98f9-aecdd23ee618","Type":"ContainerStarted","Data":"34ff7a59e906f31ff25df9bf8e9cb95f9742fc3f296c922d3f76e3f0422841c1"} Aug 04 10:31:54.867640 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:54.867586 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2mp6s" podStartSLOduration=2.374260938 podStartE2EDuration="4.86756886s" totalImagesPullingTime="2.493307922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:31:54.865926096 +0000 UTC m=+275.817333646" watchObservedRunningTime="2026-08-04 10:31:54.86756886 +0000 UTC m=+275.818976409" Aug 04 10:31:59.214365 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214325 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4bs2n"] Aug 04 10:31:59.214765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214730 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerName="util" Aug 04 10:31:59.214765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214743 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerName="util" Aug 04 10:31:59.214765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214753 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerName="pull" Aug 04 10:31:59.214765 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214759 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerName="pull" Aug 04 10:31:59.214951 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214769 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerName="extract" Aug 04 10:31:59.214951 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214775 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerName="extract" Aug 04 10:31:59.214951 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.214871 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a7727b-b0eb-4e86-a066-6bf5f6acd546" containerName="extract" Aug 04 10:31:59.218043 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.218026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.220345 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.220324 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-69nr2\"" Aug 04 10:31:59.225048 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.225024 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-4bs2n\" (UID: \"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce\") " pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.225135 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.225061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd2f\" (UniqueName: \"kubernetes.io/projected/fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce-kube-api-access-dwd2f\") pod \"cert-manager-55b4d954d5-4bs2n\" (UID: \"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce\") " pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.228261 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.228237 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4bs2n"] Aug 04 10:31:59.326469 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.326419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-4bs2n\" (UID: \"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce\") " pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.326469 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.326469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd2f\" (UniqueName: \"kubernetes.io/projected/fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce-kube-api-access-dwd2f\") pod \"cert-manager-55b4d954d5-4bs2n\" (UID: \"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce\") " pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.334643 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.334609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-4bs2n\" (UID: \"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce\") " pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.334773 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.334673 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd2f\" (UniqueName: \"kubernetes.io/projected/fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce-kube-api-access-dwd2f\") pod \"cert-manager-55b4d954d5-4bs2n\" (UID: \"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce\") " pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.527567 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.527523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4bs2n" Aug 04 10:31:59.670812 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.670780 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4bs2n"] Aug 04 10:31:59.673782 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:31:59.673743 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb1db2ac_e8d3_47a0_b57d_06af52fdf0ce.slice/crio-cccc9e04a8bf2a2d1782f715696f2cb3924ce871ffedf219632b3a00eab67dad WatchSource:0}: Error finding container cccc9e04a8bf2a2d1782f715696f2cb3924ce871ffedf219632b3a00eab67dad: Status 404 returned error can't find the container with id cccc9e04a8bf2a2d1782f715696f2cb3924ce871ffedf219632b3a00eab67dad Aug 04 10:31:59.875431 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.875333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4bs2n" event={"ID":"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce","Type":"ContainerStarted","Data":"e529125b81b715c67576672b0d407bd8a60cfae25a1dd1d01edb95214cab092b"} Aug 04 10:31:59.875431 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.875380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4bs2n" event={"ID":"fb1db2ac-e8d3-47a0-b57d-06af52fdf0ce","Type":"ContainerStarted","Data":"cccc9e04a8bf2a2d1782f715696f2cb3924ce871ffedf219632b3a00eab67dad"} Aug 04 10:31:59.892615 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:31:59.892556 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4bs2n" podStartSLOduration=0.892538442 podStartE2EDuration="892.538442ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:31:59.891785095 +0000 UTC m=+280.843192646" watchObservedRunningTime="2026-08-04 10:31:59.892538442 +0000 UTC m=+280.843945991" Aug 04 10:32:08.065298 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.065262 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr"] Aug 04 10:32:08.068960 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.068940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.071430 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.071406 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:32:08.071782 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.071761 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:32:08.071782 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.071774 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6dcj4\"" Aug 04 10:32:08.078431 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.078404 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr"] Aug 04 10:32:08.103275 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.103230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.103442 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.103334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.103442 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.103374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zslwl\" (UniqueName: \"kubernetes.io/projected/18973833-2a21-4db7-b4a0-9ee0d8d5e558-kube-api-access-zslwl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.204215 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.204174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.204424 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.204268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.204424 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.204306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zslwl\" (UniqueName: \"kubernetes.io/projected/18973833-2a21-4db7-b4a0-9ee0d8d5e558-kube-api-access-zslwl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.204689 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.204662 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.204744 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.204670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.212606 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.212579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zslwl\" (UniqueName: \"kubernetes.io/projected/18973833-2a21-4db7-b4a0-9ee0d8d5e558-kube-api-access-zslwl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.381135 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.381039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:08.516515 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.516487 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr"] Aug 04 10:32:08.519175 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:32:08.519146 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18973833_2a21_4db7_b4a0_9ee0d8d5e558.slice/crio-f9e5d74354a5b6419f96e28422b683d33fd57c35d3303a3e18121a066bc1f32b WatchSource:0}: Error finding container f9e5d74354a5b6419f96e28422b683d33fd57c35d3303a3e18121a066bc1f32b: Status 404 returned error can't find the container with id f9e5d74354a5b6419f96e28422b683d33fd57c35d3303a3e18121a066bc1f32b Aug 04 10:32:08.919248 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.919152 2577 generic.go:356] "Generic (PLEG): container finished" podID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerID="0362f2d16b6568665882ebd9f3695d38069b1825ffee2d177203cd3a45da9055" exitCode=0 Aug 04 10:32:08.919248 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.919199 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" event={"ID":"18973833-2a21-4db7-b4a0-9ee0d8d5e558","Type":"ContainerDied","Data":"0362f2d16b6568665882ebd9f3695d38069b1825ffee2d177203cd3a45da9055"} Aug 04 10:32:08.919248 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:08.919245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" event={"ID":"18973833-2a21-4db7-b4a0-9ee0d8d5e558","Type":"ContainerStarted","Data":"f9e5d74354a5b6419f96e28422b683d33fd57c35d3303a3e18121a066bc1f32b"} Aug 04 10:32:09.926048 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:09.925959 2577 generic.go:356] "Generic (PLEG): container finished" podID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerID="276fe4e5211202113fa1e021c2a05cfc71505166fa327cfa820a0dd1ff6a1c30" exitCode=0 Aug 04 10:32:09.926048 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:09.926000 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" event={"ID":"18973833-2a21-4db7-b4a0-9ee0d8d5e558","Type":"ContainerDied","Data":"276fe4e5211202113fa1e021c2a05cfc71505166fa327cfa820a0dd1ff6a1c30"} Aug 04 10:32:10.933115 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:10.933084 2577 generic.go:356] "Generic (PLEG): container finished" podID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerID="bfe8f38b0ce34164f2d8c136e6f9d31ab0942b4c506513a6505c71f9ecd4d695" exitCode=0 Aug 04 10:32:10.933456 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:10.933138 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" event={"ID":"18973833-2a21-4db7-b4a0-9ee0d8d5e558","Type":"ContainerDied","Data":"bfe8f38b0ce34164f2d8c136e6f9d31ab0942b4c506513a6505c71f9ecd4d695"} Aug 04 10:32:12.080820 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.080781 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:12.141728 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.141684 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zslwl\" (UniqueName: \"kubernetes.io/projected/18973833-2a21-4db7-b4a0-9ee0d8d5e558-kube-api-access-zslwl\") pod \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " Aug 04 10:32:12.141939 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.141748 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-util\") pod \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " Aug 04 10:32:12.141939 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.141778 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-bundle\") pod \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\" (UID: \"18973833-2a21-4db7-b4a0-9ee0d8d5e558\") " Aug 04 10:32:12.142528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.142504 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-bundle" (OuterVolumeSpecName: "bundle") pod "18973833-2a21-4db7-b4a0-9ee0d8d5e558" (UID: "18973833-2a21-4db7-b4a0-9ee0d8d5e558"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:32:12.143949 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.143921 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18973833-2a21-4db7-b4a0-9ee0d8d5e558-kube-api-access-zslwl" (OuterVolumeSpecName: "kube-api-access-zslwl") pod "18973833-2a21-4db7-b4a0-9ee0d8d5e558" (UID: "18973833-2a21-4db7-b4a0-9ee0d8d5e558"). InnerVolumeSpecName "kube-api-access-zslwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:32:12.147672 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.147649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-util" (OuterVolumeSpecName: "util") pod "18973833-2a21-4db7-b4a0-9ee0d8d5e558" (UID: "18973833-2a21-4db7-b4a0-9ee0d8d5e558"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:32:12.242494 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.242413 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zslwl\" (UniqueName: \"kubernetes.io/projected/18973833-2a21-4db7-b4a0-9ee0d8d5e558-kube-api-access-zslwl\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:32:12.242494 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.242444 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-util\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:32:12.242494 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.242454 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18973833-2a21-4db7-b4a0-9ee0d8d5e558-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:32:12.944810 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.944780 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" Aug 04 10:32:12.944810 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.944791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affw4fr" event={"ID":"18973833-2a21-4db7-b4a0-9ee0d8d5e558","Type":"ContainerDied","Data":"f9e5d74354a5b6419f96e28422b683d33fd57c35d3303a3e18121a066bc1f32b"} Aug 04 10:32:12.945037 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:12.944861 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9e5d74354a5b6419f96e28422b683d33fd57c35d3303a3e18121a066bc1f32b" Aug 04 10:32:19.485266 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:19.485232 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:32:19.491095 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:19.491068 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:32:19.495665 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:19.495646 2577 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 10:32:20.975068 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975035 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2"] Aug 04 10:32:20.977470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975408 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerName="pull" Aug 04 10:32:20.977470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975420 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerName="pull" Aug 04 10:32:20.977470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975440 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerName="util" Aug 04 10:32:20.977470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975446 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerName="util" Aug 04 10:32:20.977470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975454 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerName="extract" Aug 04 10:32:20.977470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975459 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerName="extract" Aug 04 10:32:20.977470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.975533 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18973833-2a21-4db7-b4a0-9ee0d8d5e558" containerName="extract" Aug 04 10:32:20.978357 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.978337 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:20.980642 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.980615 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 10:32:20.980762 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.980620 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 10:32:20.980972 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.980957 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 10:32:20.981182 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.981168 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 10:32:20.981242 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:20.981196 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hd694\"" Aug 04 10:32:21.000805 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.000776 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2"] Aug 04 10:32:21.127253 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.127209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc039fc3-f06b-4396-a1d6-7e7a5129a432-apiservice-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.127446 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.127265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc039fc3-f06b-4396-a1d6-7e7a5129a432-manifests\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.127446 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.127286 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc039fc3-f06b-4396-a1d6-7e7a5129a432-tmp\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.127446 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.127320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc039fc3-f06b-4396-a1d6-7e7a5129a432-webhook-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.127446 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.127373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7sm\" (UniqueName: \"kubernetes.io/projected/dc039fc3-f06b-4396-a1d6-7e7a5129a432-kube-api-access-9f7sm\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.228915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.228804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc039fc3-f06b-4396-a1d6-7e7a5129a432-apiservice-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.228915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.228880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc039fc3-f06b-4396-a1d6-7e7a5129a432-manifests\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.228915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.228904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc039fc3-f06b-4396-a1d6-7e7a5129a432-tmp\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.229185 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.228924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc039fc3-f06b-4396-a1d6-7e7a5129a432-webhook-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.229185 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.228972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7sm\" (UniqueName: \"kubernetes.io/projected/dc039fc3-f06b-4396-a1d6-7e7a5129a432-kube-api-access-9f7sm\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.229295 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.229271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc039fc3-f06b-4396-a1d6-7e7a5129a432-manifests\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.231228 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.231198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc039fc3-f06b-4396-a1d6-7e7a5129a432-tmp\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.231530 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.231506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc039fc3-f06b-4396-a1d6-7e7a5129a432-webhook-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.231599 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.231510 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc039fc3-f06b-4396-a1d6-7e7a5129a432-apiservice-cert\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.237503 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.237480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7sm\" (UniqueName: \"kubernetes.io/projected/dc039fc3-f06b-4396-a1d6-7e7a5129a432-kube-api-access-9f7sm\") pod \"opendatahub-operator-controller-manager-598d67d848-8lxk2\" (UID: \"dc039fc3-f06b-4396-a1d6-7e7a5129a432\") " pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.289644 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.289606 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:21.430381 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.430354 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2"] Aug 04 10:32:21.433053 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:32:21.433023 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc039fc3_f06b_4396_a1d6_7e7a5129a432.slice/crio-d6f3a11a93bd5e3ff08c7127adcfbc6210851685e67a57b339adb5e1fd108f4a WatchSource:0}: Error finding container d6f3a11a93bd5e3ff08c7127adcfbc6210851685e67a57b339adb5e1fd108f4a: Status 404 returned error can't find the container with id d6f3a11a93bd5e3ff08c7127adcfbc6210851685e67a57b339adb5e1fd108f4a Aug 04 10:32:21.434919 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.434903 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:32:21.990296 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:21.990259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" event={"ID":"dc039fc3-f06b-4396-a1d6-7e7a5129a432","Type":"ContainerStarted","Data":"d6f3a11a93bd5e3ff08c7127adcfbc6210851685e67a57b339adb5e1fd108f4a"} Aug 04 10:32:25.010318 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:25.010273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" event={"ID":"dc039fc3-f06b-4396-a1d6-7e7a5129a432","Type":"ContainerStarted","Data":"811e5e0442824d5da4a1f13c66a54f44517a21a001ee62c8d4bf39706a736659"} Aug 04 10:32:26.017029 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:26.016987 2577 generic.go:356] "Generic (PLEG): container finished" podID="dc039fc3-f06b-4396-a1d6-7e7a5129a432" containerID="811e5e0442824d5da4a1f13c66a54f44517a21a001ee62c8d4bf39706a736659" exitCode=0 Aug 04 10:32:26.017418 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:26.017070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" event={"ID":"dc039fc3-f06b-4396-a1d6-7e7a5129a432","Type":"ContainerDied","Data":"811e5e0442824d5da4a1f13c66a54f44517a21a001ee62c8d4bf39706a736659"} Aug 04 10:32:27.023775 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:27.023738 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" event={"ID":"dc039fc3-f06b-4396-a1d6-7e7a5129a432","Type":"ContainerStarted","Data":"e37f4975d3d85fbc906b03c0c6419d390d512e00eebcfd4feab66220fb8dfc9d"} Aug 04 10:32:27.024191 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:27.023974 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:27.047044 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:27.046981 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" podStartSLOduration=2.672366254 podStartE2EDuration="7.046962314s" totalImagesPullingTime="3.330625197s" totalInitContainerRuntime="1.043970863s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:27.045212508 +0000 UTC m=+307.996620057" watchObservedRunningTime="2026-08-04 10:32:27.046962314 +0000 UTC m=+307.998369870" Aug 04 10:32:38.032211 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:38.032175 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-598d67d848-8lxk2" Aug 04 10:32:40.270480 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.270436 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp"] Aug 04 10:32:40.279072 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.279045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.281251 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.281222 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 10:32:40.281251 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.281232 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 10:32:40.281424 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.281285 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vxqnk\"" Aug 04 10:32:40.286298 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.286271 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp"] Aug 04 10:32:40.398589 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.398552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25brt\" (UniqueName: \"kubernetes.io/projected/006ad155-43cc-4072-9e9a-b87fb1387ae8-kube-api-access-25brt\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.398775 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.398677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006ad155-43cc-4072-9e9a-b87fb1387ae8-tls-certs\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.398775 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.398711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006ad155-43cc-4072-9e9a-b87fb1387ae8-tmp\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.499766 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.499731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25brt\" (UniqueName: \"kubernetes.io/projected/006ad155-43cc-4072-9e9a-b87fb1387ae8-kube-api-access-25brt\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.499948 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.499810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006ad155-43cc-4072-9e9a-b87fb1387ae8-tls-certs\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.499948 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.499857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006ad155-43cc-4072-9e9a-b87fb1387ae8-tmp\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.502081 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.502051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006ad155-43cc-4072-9e9a-b87fb1387ae8-tmp\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.502314 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.502292 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006ad155-43cc-4072-9e9a-b87fb1387ae8-tls-certs\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.507990 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.507968 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25brt\" (UniqueName: \"kubernetes.io/projected/006ad155-43cc-4072-9e9a-b87fb1387ae8-kube-api-access-25brt\") pod \"kube-auth-proxy-5bd95bfc5-5zztp\" (UID: \"006ad155-43cc-4072-9e9a-b87fb1387ae8\") " pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.590079 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.589996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" Aug 04 10:32:40.726621 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.726596 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59766cd755-prqg4"] Aug 04 10:32:40.728524 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:32:40.728495 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006ad155_43cc_4072_9e9a_b87fb1387ae8.slice/crio-97e138dcb83ef2fb94ab505b65fc148c6bf4df87273156e29ed1c25b5d4a8218 WatchSource:0}: Error finding container 97e138dcb83ef2fb94ab505b65fc148c6bf4df87273156e29ed1c25b5d4a8218: Status 404 returned error can't find the container with id 97e138dcb83ef2fb94ab505b65fc148c6bf4df87273156e29ed1c25b5d4a8218 Aug 04 10:32:40.731611 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.731593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp"] Aug 04 10:32:40.731694 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.731679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:40.733718 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.733701 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 10:32:40.733793 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.733748 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 10:32:40.733986 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.733959 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 10:32:40.734098 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.734053 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 10:32:40.734161 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.734099 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 10:32:40.734217 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.734166 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xbfjb\"" Aug 04 10:32:40.740130 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.740111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59766cd755-prqg4"] Aug 04 10:32:40.902630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.902548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk47m\" (UniqueName: \"kubernetes.io/projected/5edc4f13-ebeb-4ff6-80bd-b791279204a3-kube-api-access-xk47m\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:40.902630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.902585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5edc4f13-ebeb-4ff6-80bd-b791279204a3-manager-config\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:40.902630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.902609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edc4f13-ebeb-4ff6-80bd-b791279204a3-cert\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:40.902878 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:40.902720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5edc4f13-ebeb-4ff6-80bd-b791279204a3-metrics-cert\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.003174 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.003131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5edc4f13-ebeb-4ff6-80bd-b791279204a3-metrics-cert\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.003380 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.003220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk47m\" (UniqueName: \"kubernetes.io/projected/5edc4f13-ebeb-4ff6-80bd-b791279204a3-kube-api-access-xk47m\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.003380 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.003254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5edc4f13-ebeb-4ff6-80bd-b791279204a3-manager-config\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.003380 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.003283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edc4f13-ebeb-4ff6-80bd-b791279204a3-cert\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.004066 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.004042 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5edc4f13-ebeb-4ff6-80bd-b791279204a3-manager-config\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.005708 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.005687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5edc4f13-ebeb-4ff6-80bd-b791279204a3-metrics-cert\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.005769 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.005744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5edc4f13-ebeb-4ff6-80bd-b791279204a3-cert\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.013250 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.013225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk47m\" (UniqueName: \"kubernetes.io/projected/5edc4f13-ebeb-4ff6-80bd-b791279204a3-kube-api-access-xk47m\") pod \"lws-controller-manager-59766cd755-prqg4\" (UID: \"5edc4f13-ebeb-4ff6-80bd-b791279204a3\") " pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.046282 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.046251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:41.096486 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.096328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" event={"ID":"006ad155-43cc-4072-9e9a-b87fb1387ae8","Type":"ContainerStarted","Data":"97e138dcb83ef2fb94ab505b65fc148c6bf4df87273156e29ed1c25b5d4a8218"} Aug 04 10:32:41.193488 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:41.193413 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59766cd755-prqg4"] Aug 04 10:32:41.197415 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:32:41.197387 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edc4f13_ebeb_4ff6_80bd_b791279204a3.slice/crio-fa8db042ccb63572a1f3d1ab344793ba4ce373b92605885c15edccf6c4e2cec6 WatchSource:0}: Error finding container fa8db042ccb63572a1f3d1ab344793ba4ce373b92605885c15edccf6c4e2cec6: Status 404 returned error can't find the container with id fa8db042ccb63572a1f3d1ab344793ba4ce373b92605885c15edccf6c4e2cec6 Aug 04 10:32:42.105452 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:42.105364 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" event={"ID":"5edc4f13-ebeb-4ff6-80bd-b791279204a3","Type":"ContainerStarted","Data":"fa8db042ccb63572a1f3d1ab344793ba4ce373b92605885c15edccf6c4e2cec6"} Aug 04 10:32:44.120772 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:44.120736 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" event={"ID":"5edc4f13-ebeb-4ff6-80bd-b791279204a3","Type":"ContainerStarted","Data":"eb48f20c174a17cbcc060f4a40b44083a855c238ddcc50809193831bc561f910"} Aug 04 10:32:44.121249 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:44.120848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:32:44.122411 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:44.122370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" event={"ID":"006ad155-43cc-4072-9e9a-b87fb1387ae8","Type":"ContainerStarted","Data":"89ed47756870a52548ec8daaabcca2f21b8ab113423dc545bbc7e0d89b8d8682"} Aug 04 10:32:44.138355 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:44.138304 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" podStartSLOduration=1.349084435 podStartE2EDuration="4.138288395s" totalImagesPullingTime="2.78920396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:44.137003683 +0000 UTC m=+325.088411233" watchObservedRunningTime="2026-08-04 10:32:44.138288395 +0000 UTC m=+325.089695944" Aug 04 10:32:44.153712 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:44.153654 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bd95bfc5-5zztp" podStartSLOduration=0.887481368 podStartE2EDuration="4.153633964s" totalImagesPullingTime="3.266152596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:32:44.15328739 +0000 UTC m=+325.104694941" watchObservedRunningTime="2026-08-04 10:32:44.153633964 +0000 UTC m=+325.105041517" Aug 04 10:32:55.129425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:32:55.129391 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59766cd755-prqg4" Aug 04 10:33:44.464101 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.464062 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5"] Aug 04 10:33:44.467841 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.467814 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.469975 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.469950 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 10:33:44.470084 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.469975 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 10:33:44.470455 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.470437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6dcj4\"" Aug 04 10:33:44.478661 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.478637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5"] Aug 04 10:33:44.561650 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.561613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.561650 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.561649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqdz\" (UniqueName: \"kubernetes.io/projected/7e58e6de-93db-4fa4-9ace-8490bc44ff93-kube-api-access-tqqdz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.561868 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.561669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.568371 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.568346 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2"] Aug 04 10:33:44.572085 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.572067 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.585017 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.584989 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2"] Aug 04 10:33:44.662544 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.662544 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqdz\" (UniqueName: \"kubernetes.io/projected/7e58e6de-93db-4fa4-9ace-8490bc44ff93-kube-api-access-tqqdz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.662771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.662771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zf59\" (UniqueName: \"kubernetes.io/projected/aa81e663-0dab-49a3-a030-d6555094244a-kube-api-access-2zf59\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.662771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.662771 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.662940 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.662940 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.662927 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.668310 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.668288 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx"] Aug 04 10:33:44.672114 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.672101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.672938 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.672915 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqdz\" (UniqueName: \"kubernetes.io/projected/7e58e6de-93db-4fa4-9ace-8490bc44ff93-kube-api-access-tqqdz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.683002 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.682980 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx"] Aug 04 10:33:44.763837 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.763806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.763981 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.763876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zf59\" (UniqueName: \"kubernetes.io/projected/aa81e663-0dab-49a3-a030-d6555094244a-kube-api-access-2zf59\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.763981 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.763936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.763981 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.763969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.764126 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.763998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.764126 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.764022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rmj5\" (UniqueName: \"kubernetes.io/projected/49d41715-556f-4b09-bde2-6c298cbf6256-kube-api-access-2rmj5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.764327 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.764306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.764385 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.764368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.772654 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.772619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zf59\" (UniqueName: \"kubernetes.io/projected/aa81e663-0dab-49a3-a030-d6555094244a-kube-api-access-2zf59\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.777258 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.777238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:44.790732 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.790708 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp"] Aug 04 10:33:44.794785 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.794763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.804363 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.804338 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp"] Aug 04 10:33:44.865115 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.865115 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rmj5\" (UniqueName: \"kubernetes.io/projected/49d41715-556f-4b09-bde2-6c298cbf6256-kube-api-access-2rmj5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.865316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.865316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.865316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.865316 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865280 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtcg\" (UniqueName: \"kubernetes.io/projected/2fcb7f7e-bba7-47ae-a9cf-320841c95046-kube-api-access-hvtcg\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.865532 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865503 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.865967 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.865945 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.873786 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.873763 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rmj5\" (UniqueName: \"kubernetes.io/projected/49d41715-556f-4b09-bde2-6c298cbf6256-kube-api-access-2rmj5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:44.881646 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.881612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:44.919604 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.919578 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5"] Aug 04 10:33:44.921118 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:33:44.921077 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e58e6de_93db_4fa4_9ace_8490bc44ff93.slice/crio-4979c30b6a6a25efd03d4e730b8e09af69f82633ef62b3a74425dad11443b237 WatchSource:0}: Error finding container 4979c30b6a6a25efd03d4e730b8e09af69f82633ef62b3a74425dad11443b237: Status 404 returned error can't find the container with id 4979c30b6a6a25efd03d4e730b8e09af69f82633ef62b3a74425dad11443b237 Aug 04 10:33:44.965773 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.965749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.965890 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.965785 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.965890 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.965815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtcg\" (UniqueName: \"kubernetes.io/projected/2fcb7f7e-bba7-47ae-a9cf-320841c95046-kube-api-access-hvtcg\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.966206 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.966181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.966278 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.966256 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.974506 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.974482 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtcg\" (UniqueName: \"kubernetes.io/projected/2fcb7f7e-bba7-47ae-a9cf-320841c95046-kube-api-access-hvtcg\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:44.989585 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:44.989562 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:45.032194 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.031672 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2"] Aug 04 10:33:45.034202 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:33:45.034174 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa81e663_0dab_49a3_a030_d6555094244a.slice/crio-e2d65c41fb34d7a267b3d930b624feaa27ee6b71a10df3eb8753167636c24b3b WatchSource:0}: Error finding container e2d65c41fb34d7a267b3d930b624feaa27ee6b71a10df3eb8753167636c24b3b: Status 404 returned error can't find the container with id e2d65c41fb34d7a267b3d930b624feaa27ee6b71a10df3eb8753167636c24b3b Aug 04 10:33:45.111872 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.111820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:45.133745 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.133720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx"] Aug 04 10:33:45.136288 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:33:45.136262 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d41715_556f_4b09_bde2_6c298cbf6256.slice/crio-4b6a7c899bd7f6adc071a4bd6f95a6b670f67760069d330c79744558570932e0 WatchSource:0}: Error finding container 4b6a7c899bd7f6adc071a4bd6f95a6b670f67760069d330c79744558570932e0: Status 404 returned error can't find the container with id 4b6a7c899bd7f6adc071a4bd6f95a6b670f67760069d330c79744558570932e0 Aug 04 10:33:45.268304 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.268278 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp"] Aug 04 10:33:45.293667 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:33:45.293613 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcb7f7e_bba7_47ae_a9cf_320841c95046.slice/crio-d0cb8180af9c0d5561e3080346e77cfbe0ff2986a01bf08908e005551192b938 WatchSource:0}: Error finding container d0cb8180af9c0d5561e3080346e77cfbe0ff2986a01bf08908e005551192b938: Status 404 returned error can't find the container with id d0cb8180af9c0d5561e3080346e77cfbe0ff2986a01bf08908e005551192b938 Aug 04 10:33:45.436703 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.436671 2577 generic.go:356] "Generic (PLEG): container finished" podID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerID="da38ff790a67b9608cf1027995d8ecdbaab1088154c1ccd2e251f58e2718dcbd" exitCode=0 Aug 04 10:33:45.436857 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.436757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" event={"ID":"7e58e6de-93db-4fa4-9ace-8490bc44ff93","Type":"ContainerDied","Data":"da38ff790a67b9608cf1027995d8ecdbaab1088154c1ccd2e251f58e2718dcbd"} Aug 04 10:33:45.436857 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.436816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" event={"ID":"7e58e6de-93db-4fa4-9ace-8490bc44ff93","Type":"ContainerStarted","Data":"4979c30b6a6a25efd03d4e730b8e09af69f82633ef62b3a74425dad11443b237"} Aug 04 10:33:45.438506 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.438479 2577 generic.go:356] "Generic (PLEG): container finished" podID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerID="8ec66c09c85c283c3fe0637055fc68bee786affed5abc4683529677e6f60cab2" exitCode=0 Aug 04 10:33:45.438612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.438559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" event={"ID":"2fcb7f7e-bba7-47ae-a9cf-320841c95046","Type":"ContainerDied","Data":"8ec66c09c85c283c3fe0637055fc68bee786affed5abc4683529677e6f60cab2"} Aug 04 10:33:45.438612 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.438605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" event={"ID":"2fcb7f7e-bba7-47ae-a9cf-320841c95046","Type":"ContainerStarted","Data":"d0cb8180af9c0d5561e3080346e77cfbe0ff2986a01bf08908e005551192b938"} Aug 04 10:33:45.440426 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.440404 2577 generic.go:356] "Generic (PLEG): container finished" podID="49d41715-556f-4b09-bde2-6c298cbf6256" containerID="2e8bebcb9da450b1eedbc9c9131aec7266f44e8065617969d99161d5190735e7" exitCode=0 Aug 04 10:33:45.440527 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.440438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" event={"ID":"49d41715-556f-4b09-bde2-6c298cbf6256","Type":"ContainerDied","Data":"2e8bebcb9da450b1eedbc9c9131aec7266f44e8065617969d99161d5190735e7"} Aug 04 10:33:45.440527 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.440475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" event={"ID":"49d41715-556f-4b09-bde2-6c298cbf6256","Type":"ContainerStarted","Data":"4b6a7c899bd7f6adc071a4bd6f95a6b670f67760069d330c79744558570932e0"} Aug 04 10:33:45.442251 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.442232 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa81e663-0dab-49a3-a030-d6555094244a" containerID="c401f8d8f6a07872047ac47ea969181689685bbc00e3879c98db1c69223a95a5" exitCode=0 Aug 04 10:33:45.442327 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.442265 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" event={"ID":"aa81e663-0dab-49a3-a030-d6555094244a","Type":"ContainerDied","Data":"c401f8d8f6a07872047ac47ea969181689685bbc00e3879c98db1c69223a95a5"} Aug 04 10:33:45.442327 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:45.442301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" event={"ID":"aa81e663-0dab-49a3-a030-d6555094244a","Type":"ContainerStarted","Data":"e2d65c41fb34d7a267b3d930b624feaa27ee6b71a10df3eb8753167636c24b3b"} Aug 04 10:33:46.454346 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:46.454319 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa81e663-0dab-49a3-a030-d6555094244a" containerID="8d8af6a916539bf4408299a535edabb7fc8c02643fce5fa201966169044595da" exitCode=0 Aug 04 10:33:46.454705 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:46.454395 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" event={"ID":"aa81e663-0dab-49a3-a030-d6555094244a","Type":"ContainerDied","Data":"8d8af6a916539bf4408299a535edabb7fc8c02643fce5fa201966169044595da"} Aug 04 10:33:46.456961 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:46.456939 2577 generic.go:356] "Generic (PLEG): container finished" podID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerID="5b518b3de83a5819d2cd5cbae227e182cdace9ac74837c83a318736ffdbbc2cf" exitCode=0 Aug 04 10:33:46.457081 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:46.456998 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" event={"ID":"7e58e6de-93db-4fa4-9ace-8490bc44ff93","Type":"ContainerDied","Data":"5b518b3de83a5819d2cd5cbae227e182cdace9ac74837c83a318736ffdbbc2cf"} Aug 04 10:33:47.464667 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.464624 2577 generic.go:356] "Generic (PLEG): container finished" podID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerID="97a55a706bb06b3aeace9e33795bb4ee5cb5e389c02e979e5a9de789626cd843" exitCode=0 Aug 04 10:33:47.465101 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.464704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" event={"ID":"2fcb7f7e-bba7-47ae-a9cf-320841c95046","Type":"ContainerDied","Data":"97a55a706bb06b3aeace9e33795bb4ee5cb5e389c02e979e5a9de789626cd843"} Aug 04 10:33:47.466593 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.466571 2577 generic.go:356] "Generic (PLEG): container finished" podID="49d41715-556f-4b09-bde2-6c298cbf6256" containerID="122bce1ea802a2dc48e6dd4220a883c24b529d7ef7b1391b45b51b7eb0f6ffc6" exitCode=0 Aug 04 10:33:47.466690 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.466653 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" event={"ID":"49d41715-556f-4b09-bde2-6c298cbf6256","Type":"ContainerDied","Data":"122bce1ea802a2dc48e6dd4220a883c24b529d7ef7b1391b45b51b7eb0f6ffc6"} Aug 04 10:33:47.468896 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.468877 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa81e663-0dab-49a3-a030-d6555094244a" containerID="87365fa43c83b597f5b4e2bdddedc6a718a1ce3d9033a322722c4e0a2bdc5b09" exitCode=0 Aug 04 10:33:47.468999 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.468903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" event={"ID":"aa81e663-0dab-49a3-a030-d6555094244a","Type":"ContainerDied","Data":"87365fa43c83b597f5b4e2bdddedc6a718a1ce3d9033a322722c4e0a2bdc5b09"} Aug 04 10:33:47.471087 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.471056 2577 generic.go:356] "Generic (PLEG): container finished" podID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerID="0a9cf86e006553cdc4ec63e269a5c890df242581f04dbdf2e78485d147377537" exitCode=0 Aug 04 10:33:47.471180 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:47.471123 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" event={"ID":"7e58e6de-93db-4fa4-9ace-8490bc44ff93","Type":"ContainerDied","Data":"0a9cf86e006553cdc4ec63e269a5c890df242581f04dbdf2e78485d147377537"} Aug 04 10:33:48.478191 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.478160 2577 generic.go:356] "Generic (PLEG): container finished" podID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerID="5b8c8a73055b04ee7d1da8a3f873f43fee491265f3d2ee10beaa8f3c2cf7e4ee" exitCode=0 Aug 04 10:33:48.478667 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.478231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" event={"ID":"2fcb7f7e-bba7-47ae-a9cf-320841c95046","Type":"ContainerDied","Data":"5b8c8a73055b04ee7d1da8a3f873f43fee491265f3d2ee10beaa8f3c2cf7e4ee"} Aug 04 10:33:48.480295 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.480269 2577 generic.go:356] "Generic (PLEG): container finished" podID="49d41715-556f-4b09-bde2-6c298cbf6256" containerID="c87360075a1b01c30a1e149838d26e03528b4205511b3cb1e176d1cc4d59bf29" exitCode=0 Aug 04 10:33:48.480397 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.480350 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" event={"ID":"49d41715-556f-4b09-bde2-6c298cbf6256","Type":"ContainerDied","Data":"c87360075a1b01c30a1e149838d26e03528b4205511b3cb1e176d1cc4d59bf29"} Aug 04 10:33:48.620845 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.620811 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:48.650723 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.650697 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:48.693501 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.693460 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-bundle\") pod \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " Aug 04 10:33:48.693696 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.693525 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqqdz\" (UniqueName: \"kubernetes.io/projected/7e58e6de-93db-4fa4-9ace-8490bc44ff93-kube-api-access-tqqdz\") pod \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " Aug 04 10:33:48.693696 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.693636 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-util\") pod \"aa81e663-0dab-49a3-a030-d6555094244a\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " Aug 04 10:33:48.693696 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.693662 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-bundle\") pod \"aa81e663-0dab-49a3-a030-d6555094244a\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " Aug 04 10:33:48.693889 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.693698 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zf59\" (UniqueName: \"kubernetes.io/projected/aa81e663-0dab-49a3-a030-d6555094244a-kube-api-access-2zf59\") pod \"aa81e663-0dab-49a3-a030-d6555094244a\" (UID: \"aa81e663-0dab-49a3-a030-d6555094244a\") " Aug 04 10:33:48.693889 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.693747 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-util\") pod \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\" (UID: \"7e58e6de-93db-4fa4-9ace-8490bc44ff93\") " Aug 04 10:33:48.694256 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.694228 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-bundle" (OuterVolumeSpecName: "bundle") pod "aa81e663-0dab-49a3-a030-d6555094244a" (UID: "aa81e663-0dab-49a3-a030-d6555094244a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:48.694409 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.694381 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-bundle" (OuterVolumeSpecName: "bundle") pod "7e58e6de-93db-4fa4-9ace-8490bc44ff93" (UID: "7e58e6de-93db-4fa4-9ace-8490bc44ff93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:48.695862 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.695807 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58e6de-93db-4fa4-9ace-8490bc44ff93-kube-api-access-tqqdz" (OuterVolumeSpecName: "kube-api-access-tqqdz") pod "7e58e6de-93db-4fa4-9ace-8490bc44ff93" (UID: "7e58e6de-93db-4fa4-9ace-8490bc44ff93"). InnerVolumeSpecName "kube-api-access-tqqdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:33:48.695964 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.695873 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa81e663-0dab-49a3-a030-d6555094244a-kube-api-access-2zf59" (OuterVolumeSpecName: "kube-api-access-2zf59") pod "aa81e663-0dab-49a3-a030-d6555094244a" (UID: "aa81e663-0dab-49a3-a030-d6555094244a"). InnerVolumeSpecName "kube-api-access-2zf59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:33:48.702137 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.702109 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-util" (OuterVolumeSpecName: "util") pod "7e58e6de-93db-4fa4-9ace-8490bc44ff93" (UID: "7e58e6de-93db-4fa4-9ace-8490bc44ff93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:48.702242 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.702224 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-util" (OuterVolumeSpecName: "util") pod "aa81e663-0dab-49a3-a030-d6555094244a" (UID: "aa81e663-0dab-49a3-a030-d6555094244a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:48.794630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.794596 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqqdz\" (UniqueName: \"kubernetes.io/projected/7e58e6de-93db-4fa4-9ace-8490bc44ff93-kube-api-access-tqqdz\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:48.794630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.794623 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-util\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:48.794630 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.794634 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa81e663-0dab-49a3-a030-d6555094244a-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:48.794878 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.794642 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zf59\" (UniqueName: \"kubernetes.io/projected/aa81e663-0dab-49a3-a030-d6555094244a-kube-api-access-2zf59\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:48.794878 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.794651 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-util\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:48.794878 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:48.794659 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6de-93db-4fa4-9ace-8490bc44ff93-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:49.488178 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.488146 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" Aug 04 10:33:49.488633 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.488145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrlrq2" event={"ID":"aa81e663-0dab-49a3-a030-d6555094244a","Type":"ContainerDied","Data":"e2d65c41fb34d7a267b3d930b624feaa27ee6b71a10df3eb8753167636c24b3b"} Aug 04 10:33:49.488633 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.488284 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2d65c41fb34d7a267b3d930b624feaa27ee6b71a10df3eb8753167636c24b3b" Aug 04 10:33:49.490209 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.490190 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" Aug 04 10:33:49.490318 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.490229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cs5dt5" event={"ID":"7e58e6de-93db-4fa4-9ace-8490bc44ff93","Type":"ContainerDied","Data":"4979c30b6a6a25efd03d4e730b8e09af69f82633ef62b3a74425dad11443b237"} Aug 04 10:33:49.490318 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.490286 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4979c30b6a6a25efd03d4e730b8e09af69f82633ef62b3a74425dad11443b237" Aug 04 10:33:49.647721 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.647696 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:49.666738 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.666712 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:49.700879 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.700851 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-util\") pod \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " Aug 04 10:33:49.701054 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.700895 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvtcg\" (UniqueName: \"kubernetes.io/projected/2fcb7f7e-bba7-47ae-a9cf-320841c95046-kube-api-access-hvtcg\") pod \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " Aug 04 10:33:49.701054 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.700935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rmj5\" (UniqueName: \"kubernetes.io/projected/49d41715-556f-4b09-bde2-6c298cbf6256-kube-api-access-2rmj5\") pod \"49d41715-556f-4b09-bde2-6c298cbf6256\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " Aug 04 10:33:49.701054 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.700952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-util\") pod \"49d41715-556f-4b09-bde2-6c298cbf6256\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " Aug 04 10:33:49.701054 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.700970 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-bundle\") pod \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\" (UID: \"2fcb7f7e-bba7-47ae-a9cf-320841c95046\") " Aug 04 10:33:49.701054 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.701000 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-bundle\") pod \"49d41715-556f-4b09-bde2-6c298cbf6256\" (UID: \"49d41715-556f-4b09-bde2-6c298cbf6256\") " Aug 04 10:33:49.701690 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.701660 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-bundle" (OuterVolumeSpecName: "bundle") pod "49d41715-556f-4b09-bde2-6c298cbf6256" (UID: "49d41715-556f-4b09-bde2-6c298cbf6256"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:49.701870 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.701798 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-bundle" (OuterVolumeSpecName: "bundle") pod "2fcb7f7e-bba7-47ae-a9cf-320841c95046" (UID: "2fcb7f7e-bba7-47ae-a9cf-320841c95046"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:49.703731 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.703707 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d41715-556f-4b09-bde2-6c298cbf6256-kube-api-access-2rmj5" (OuterVolumeSpecName: "kube-api-access-2rmj5") pod "49d41715-556f-4b09-bde2-6c298cbf6256" (UID: "49d41715-556f-4b09-bde2-6c298cbf6256"). InnerVolumeSpecName "kube-api-access-2rmj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:33:49.703804 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.703719 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcb7f7e-bba7-47ae-a9cf-320841c95046-kube-api-access-hvtcg" (OuterVolumeSpecName: "kube-api-access-hvtcg") pod "2fcb7f7e-bba7-47ae-a9cf-320841c95046" (UID: "2fcb7f7e-bba7-47ae-a9cf-320841c95046"). InnerVolumeSpecName "kube-api-access-hvtcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:33:49.709923 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.709877 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-util" (OuterVolumeSpecName: "util") pod "49d41715-556f-4b09-bde2-6c298cbf6256" (UID: "49d41715-556f-4b09-bde2-6c298cbf6256"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:49.710207 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.710190 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-util" (OuterVolumeSpecName: "util") pod "2fcb7f7e-bba7-47ae-a9cf-320841c95046" (UID: "2fcb7f7e-bba7-47ae-a9cf-320841c95046"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 10:33:49.802342 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.802313 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-util\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:49.802342 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.802341 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvtcg\" (UniqueName: \"kubernetes.io/projected/2fcb7f7e-bba7-47ae-a9cf-320841c95046-kube-api-access-hvtcg\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:49.802520 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.802352 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rmj5\" (UniqueName: \"kubernetes.io/projected/49d41715-556f-4b09-bde2-6c298cbf6256-kube-api-access-2rmj5\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:49.802520 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.802361 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-util\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:49.802520 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.802369 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2fcb7f7e-bba7-47ae-a9cf-320841c95046-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:49.802520 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:49.802376 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49d41715-556f-4b09-bde2-6c298cbf6256-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:33:50.497297 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:50.497263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" event={"ID":"2fcb7f7e-bba7-47ae-a9cf-320841c95046","Type":"ContainerDied","Data":"d0cb8180af9c0d5561e3080346e77cfbe0ff2986a01bf08908e005551192b938"} Aug 04 10:33:50.497787 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:50.497275 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870532qsrp" Aug 04 10:33:50.497787 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:50.497351 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0cb8180af9c0d5561e3080346e77cfbe0ff2986a01bf08908e005551192b938" Aug 04 10:33:50.499459 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:50.499432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" event={"ID":"49d41715-556f-4b09-bde2-6c298cbf6256","Type":"ContainerDied","Data":"4b6a7c899bd7f6adc071a4bd6f95a6b670f67760069d330c79744558570932e0"} Aug 04 10:33:50.499586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:50.499479 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b6a7c899bd7f6adc071a4bd6f95a6b670f67760069d330c79744558570932e0" Aug 04 10:33:50.499586 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:33:50.499483 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a09d2qx" Aug 04 10:33:50.521631 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:33:50.521602 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcb7f7e_bba7_47ae_a9cf_320841c95046.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d41715_556f_4b09_bde2_6c298cbf6256.slice\": RecentStats: unable to find data in memory cache]" Aug 04 10:34:10.210792 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.210753 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98"] Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211134 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerName="extract" Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211146 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerName="extract" Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211161 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa81e663-0dab-49a3-a030-d6555094244a" containerName="util" Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211167 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa81e663-0dab-49a3-a030-d6555094244a" containerName="util" Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211173 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerName="pull" Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211179 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerName="pull" Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211186 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa81e663-0dab-49a3-a030-d6555094244a" containerName="extract" Aug 04 10:34:10.211186 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211191 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa81e663-0dab-49a3-a030-d6555094244a" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211200 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerName="util" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211205 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerName="util" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211212 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerName="util" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211217 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerName="util" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211225 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa81e663-0dab-49a3-a030-d6555094244a" containerName="pull" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211230 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa81e663-0dab-49a3-a030-d6555094244a" containerName="pull" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211235 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d41715-556f-4b09-bde2-6c298cbf6256" containerName="pull" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211239 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d41715-556f-4b09-bde2-6c298cbf6256" containerName="pull" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211244 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d41715-556f-4b09-bde2-6c298cbf6256" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211249 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d41715-556f-4b09-bde2-6c298cbf6256" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211266 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerName="pull" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211272 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerName="pull" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211277 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211282 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211294 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d41715-556f-4b09-bde2-6c298cbf6256" containerName="util" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211300 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d41715-556f-4b09-bde2-6c298cbf6256" containerName="util" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211368 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa81e663-0dab-49a3-a030-d6555094244a" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211376 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e58e6de-93db-4fa4-9ace-8490bc44ff93" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211384 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fcb7f7e-bba7-47ae-a9cf-320841c95046" containerName="extract" Aug 04 10:34:10.211436 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.211389 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d41715-556f-4b09-bde2-6c298cbf6256" containerName="extract" Aug 04 10:34:10.216165 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.216142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.218336 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.218310 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 04 10:34:10.218468 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.218369 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 10:34:10.218468 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.218369 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 10:34:10.218468 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.218414 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 04 10:34:10.218735 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.218722 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r7drb\"" Aug 04 10:34:10.225641 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.225618 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98"] Aug 04 10:34:10.281615 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.281584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.281799 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.281628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.281799 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.281718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6z57\" (UniqueName: \"kubernetes.io/projected/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-kube-api-access-l6z57\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.382987 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.382950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.383169 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.383000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.383169 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.383057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6z57\" (UniqueName: \"kubernetes.io/projected/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-kube-api-access-l6z57\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.383169 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:34:10.383097 2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Aug 04 10:34:10.383338 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:34:10.383179 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-plugin-serving-cert podName:27aecda2-b2c7-4e72-9caa-3c80cafa8bac nodeName:}" failed. No retries permitted until 2026-08-04 10:34:10.883162175 +0000 UTC m=+411.834569703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-4nv98" (UID: "27aecda2-b2c7-4e72-9caa-3c80cafa8bac") : secret "plugin-serving-cert" not found Aug 04 10:34:10.383724 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.383699 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.393128 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.393099 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6z57\" (UniqueName: \"kubernetes.io/projected/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-kube-api-access-l6z57\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.887530 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.887494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:10.889915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:10.889886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27aecda2-b2c7-4e72-9caa-3c80cafa8bac-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-4nv98\" (UID: \"27aecda2-b2c7-4e72-9caa-3c80cafa8bac\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:11.126870 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:11.126812 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" Aug 04 10:34:11.302347 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:11.302322 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98"] Aug 04 10:34:11.303645 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:34:11.303616 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27aecda2_b2c7_4e72_9caa_3c80cafa8bac.slice/crio-add57d5ae570bdb49bafa12191ece33acef14ea6eb9e79efa01baaf0f127b30c WatchSource:0}: Error finding container add57d5ae570bdb49bafa12191ece33acef14ea6eb9e79efa01baaf0f127b30c: Status 404 returned error can't find the container with id add57d5ae570bdb49bafa12191ece33acef14ea6eb9e79efa01baaf0f127b30c Aug 04 10:34:11.608294 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:11.608260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" event={"ID":"27aecda2-b2c7-4e72-9caa-3c80cafa8bac","Type":"ContainerStarted","Data":"add57d5ae570bdb49bafa12191ece33acef14ea6eb9e79efa01baaf0f127b30c"} Aug 04 10:34:24.681217 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:24.681178 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" event={"ID":"27aecda2-b2c7-4e72-9caa-3c80cafa8bac","Type":"ContainerStarted","Data":"9b01a528408612e02a72c1fa373ec403a97046373b4d648bd9ab953a152eda84"} Aug 04 10:34:24.701102 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:24.701051 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-4nv98" podStartSLOduration=2.07333064 podStartE2EDuration="14.701035982s" totalImagesPullingTime="12.627705342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:34:24.699247887 +0000 UTC m=+425.650655442" watchObservedRunningTime="2026-08-04 10:34:24.701035982 +0000 UTC m=+425.652443531" Aug 04 10:34:30.804429 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:30.804393 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75cdf679f5-7vhpm"] Aug 04 10:34:55.824986 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:55.824877 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75cdf679f5-7vhpm" podUID="b3aac9f2-7c28-4e59-9b8f-63792b08ca49" containerName="console" containerID="cri-o://11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869" gracePeriod=15 Aug 04 10:34:56.071516 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.071490 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:34:56.204020 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.203928 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s4l7\" (UniqueName: \"kubernetes.io/projected/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-kube-api-access-4s4l7\") pod \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " Aug 04 10:34:56.204020 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.203969 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-oauth-serving-cert\") pod \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " Aug 04 10:34:56.204020 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.203997 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-oauth-config\") pod \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " Aug 04 10:34:56.204307 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-config\") pod \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " Aug 04 10:34:56.204307 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204060 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-service-ca\") pod \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " Aug 04 10:34:56.204307 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204090 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-serving-cert\") pod \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " Aug 04 10:34:56.204307 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204130 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-trusted-ca-bundle\") pod \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\" (UID: \"b3aac9f2-7c28-4e59-9b8f-63792b08ca49\") " Aug 04 10:34:56.204517 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204483 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b3aac9f2-7c28-4e59-9b8f-63792b08ca49" (UID: "b3aac9f2-7c28-4e59-9b8f-63792b08ca49"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:34:56.204560 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-config" (OuterVolumeSpecName: "console-config") pod "b3aac9f2-7c28-4e59-9b8f-63792b08ca49" (UID: "b3aac9f2-7c28-4e59-9b8f-63792b08ca49"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:34:56.204560 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204542 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-service-ca" (OuterVolumeSpecName: "service-ca") pod "b3aac9f2-7c28-4e59-9b8f-63792b08ca49" (UID: "b3aac9f2-7c28-4e59-9b8f-63792b08ca49"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:34:56.204794 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.204769 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b3aac9f2-7c28-4e59-9b8f-63792b08ca49" (UID: "b3aac9f2-7c28-4e59-9b8f-63792b08ca49"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:34:56.206230 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.206208 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b3aac9f2-7c28-4e59-9b8f-63792b08ca49" (UID: "b3aac9f2-7c28-4e59-9b8f-63792b08ca49"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:34:56.206416 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.206393 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b3aac9f2-7c28-4e59-9b8f-63792b08ca49" (UID: "b3aac9f2-7c28-4e59-9b8f-63792b08ca49"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:34:56.206496 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.206474 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-kube-api-access-4s4l7" (OuterVolumeSpecName: "kube-api-access-4s4l7") pod "b3aac9f2-7c28-4e59-9b8f-63792b08ca49" (UID: "b3aac9f2-7c28-4e59-9b8f-63792b08ca49"). InnerVolumeSpecName "kube-api-access-4s4l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:34:56.305473 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.305434 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-config\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:34:56.305473 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.305467 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:34:56.305473 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.305476 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-serving-cert\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:34:56.305701 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.305485 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-trusted-ca-bundle\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:34:56.305701 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.305504 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s4l7\" (UniqueName: \"kubernetes.io/projected/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-kube-api-access-4s4l7\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:34:56.305701 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.305513 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-oauth-serving-cert\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:34:56.305701 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.305522 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3aac9f2-7c28-4e59-9b8f-63792b08ca49-console-oauth-config\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:34:56.854162 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.854128 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3aac9f2-7c28-4e59-9b8f-63792b08ca49" containerID="11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869" exitCode=0 Aug 04 10:34:56.854631 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.854180 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cdf679f5-7vhpm" Aug 04 10:34:56.854631 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.854211 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cdf679f5-7vhpm" event={"ID":"b3aac9f2-7c28-4e59-9b8f-63792b08ca49","Type":"ContainerDied","Data":"11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869"} Aug 04 10:34:56.854631 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.854260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cdf679f5-7vhpm" event={"ID":"b3aac9f2-7c28-4e59-9b8f-63792b08ca49","Type":"ContainerDied","Data":"84dda9b3b5f496f20674638c02d3bf1cedf2c0de9a998897afe0e29c3d8468b1"} Aug 04 10:34:56.854631 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.854279 2577 scope.go:117] "RemoveContainer" containerID="11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869" Aug 04 10:34:56.864329 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.864313 2577 scope.go:117] "RemoveContainer" containerID="11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869" Aug 04 10:34:56.864565 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:34:56.864547 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869\": container with ID starting with 11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869 not found: ID does not exist" containerID="11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869" Aug 04 10:34:56.864619 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.864576 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869"} err="failed to get container status \"11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869\": rpc error: code = NotFound desc = could not find container \"11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869\": container with ID starting with 11eee730f853fa0050d4df56724395f65dca651b56b7f884aa40e2ab05703869 not found: ID does not exist" Aug 04 10:34:56.876660 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.876638 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75cdf679f5-7vhpm"] Aug 04 10:34:56.879969 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:56.879947 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75cdf679f5-7vhpm"] Aug 04 10:34:57.533202 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:34:57.533160 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3aac9f2-7c28-4e59-9b8f-63792b08ca49" path="/var/lib/kubelet/pods/b3aac9f2-7c28-4e59-9b8f-63792b08ca49/volumes" Aug 04 10:35:04.608188 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.608153 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-q9xgz"] Aug 04 10:35:04.608648 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.608504 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3aac9f2-7c28-4e59-9b8f-63792b08ca49" containerName="console" Aug 04 10:35:04.608648 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.608514 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3aac9f2-7c28-4e59-9b8f-63792b08ca49" containerName="console" Aug 04 10:35:04.608648 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.608593 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3aac9f2-7c28-4e59-9b8f-63792b08ca49" containerName="console" Aug 04 10:35:04.613156 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.613135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-q9xgz" Aug 04 10:35:04.615196 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.615170 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-snb4m\"" Aug 04 10:35:04.618450 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.618426 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-q9xgz"] Aug 04 10:35:04.683208 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.683163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv487\" (UniqueName: \"kubernetes.io/projected/af53f0dd-ff73-4a4f-aa1b-adf866956e64-kube-api-access-xv487\") pod \"authorino-b54df94b4-q9xgz\" (UID: \"af53f0dd-ff73-4a4f-aa1b-adf866956e64\") " pod="kuadrant-system/authorino-b54df94b4-q9xgz" Aug 04 10:35:04.772470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.772437 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lvpdb"] Aug 04 10:35:04.776041 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.776018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" Aug 04 10:35:04.784552 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.784521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv487\" (UniqueName: \"kubernetes.io/projected/af53f0dd-ff73-4a4f-aa1b-adf866956e64-kube-api-access-xv487\") pod \"authorino-b54df94b4-q9xgz\" (UID: \"af53f0dd-ff73-4a4f-aa1b-adf866956e64\") " pod="kuadrant-system/authorino-b54df94b4-q9xgz" Aug 04 10:35:04.785426 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.785407 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lvpdb"] Aug 04 10:35:04.796003 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.795980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv487\" (UniqueName: \"kubernetes.io/projected/af53f0dd-ff73-4a4f-aa1b-adf866956e64-kube-api-access-xv487\") pod \"authorino-b54df94b4-q9xgz\" (UID: \"af53f0dd-ff73-4a4f-aa1b-adf866956e64\") " pod="kuadrant-system/authorino-b54df94b4-q9xgz" Aug 04 10:35:04.886101 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.886019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r98g\" (UniqueName: \"kubernetes.io/projected/982a7c26-05e2-4a8c-9432-c5f4cd89463a-kube-api-access-9r98g\") pod \"authorino-c548b5d7d-lvpdb\" (UID: \"982a7c26-05e2-4a8c-9432-c5f4cd89463a\") " pod="kuadrant-system/authorino-c548b5d7d-lvpdb" Aug 04 10:35:04.923721 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.923687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-q9xgz" Aug 04 10:35:04.987238 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.987202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r98g\" (UniqueName: \"kubernetes.io/projected/982a7c26-05e2-4a8c-9432-c5f4cd89463a-kube-api-access-9r98g\") pod \"authorino-c548b5d7d-lvpdb\" (UID: \"982a7c26-05e2-4a8c-9432-c5f4cd89463a\") " pod="kuadrant-system/authorino-c548b5d7d-lvpdb" Aug 04 10:35:04.996584 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:04.996537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r98g\" (UniqueName: \"kubernetes.io/projected/982a7c26-05e2-4a8c-9432-c5f4cd89463a-kube-api-access-9r98g\") pod \"authorino-c548b5d7d-lvpdb\" (UID: \"982a7c26-05e2-4a8c-9432-c5f4cd89463a\") " pod="kuadrant-system/authorino-c548b5d7d-lvpdb" Aug 04 10:35:05.061411 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:05.061241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-q9xgz"] Aug 04 10:35:05.064305 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:35:05.064270 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf53f0dd_ff73_4a4f_aa1b_adf866956e64.slice/crio-cf7699fe642f63ca2116c131725b8517a6dd1d7c63d72ff687804e55346fd814 WatchSource:0}: Error finding container cf7699fe642f63ca2116c131725b8517a6dd1d7c63d72ff687804e55346fd814: Status 404 returned error can't find the container with id cf7699fe642f63ca2116c131725b8517a6dd1d7c63d72ff687804e55346fd814 Aug 04 10:35:05.085262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:05.085240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" Aug 04 10:35:05.220643 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:05.220617 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lvpdb"] Aug 04 10:35:05.223114 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:35:05.223092 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod982a7c26_05e2_4a8c_9432_c5f4cd89463a.slice/crio-ff825ccdcf2b041e7b2c5ad5b9853278ffbb8c42a45161ee37fb28b986855296 WatchSource:0}: Error finding container ff825ccdcf2b041e7b2c5ad5b9853278ffbb8c42a45161ee37fb28b986855296: Status 404 returned error can't find the container with id ff825ccdcf2b041e7b2c5ad5b9853278ffbb8c42a45161ee37fb28b986855296 Aug 04 10:35:05.903471 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:05.903397 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" event={"ID":"982a7c26-05e2-4a8c-9432-c5f4cd89463a","Type":"ContainerStarted","Data":"ff825ccdcf2b041e7b2c5ad5b9853278ffbb8c42a45161ee37fb28b986855296"} Aug 04 10:35:05.904895 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:05.904864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-q9xgz" event={"ID":"af53f0dd-ff73-4a4f-aa1b-adf866956e64","Type":"ContainerStarted","Data":"cf7699fe642f63ca2116c131725b8517a6dd1d7c63d72ff687804e55346fd814"} Aug 04 10:35:08.926346 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:08.926310 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" event={"ID":"982a7c26-05e2-4a8c-9432-c5f4cd89463a","Type":"ContainerStarted","Data":"3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28"} Aug 04 10:35:08.927867 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:08.927816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-q9xgz" event={"ID":"af53f0dd-ff73-4a4f-aa1b-adf866956e64","Type":"ContainerStarted","Data":"b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1"} Aug 04 10:35:08.942748 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:08.942690 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" podStartSLOduration=2.287688495 podStartE2EDuration="4.942674218s" totalImagesPullingTime="2.654985723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:35:08.942565862 +0000 UTC m=+469.893973411" watchObservedRunningTime="2026-08-04 10:35:08.942674218 +0000 UTC m=+469.894081768" Aug 04 10:35:08.961750 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:08.961697 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-q9xgz" podStartSLOduration=2.136585437 podStartE2EDuration="4.961683997s" totalImagesPullingTime="2.82509856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:35:08.958702747 +0000 UTC m=+469.910110300" watchObservedRunningTime="2026-08-04 10:35:08.961683997 +0000 UTC m=+469.913091546" Aug 04 10:35:09.004883 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:09.004856 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-q9xgz"] Aug 04 10:35:10.938873 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:10.938800 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-q9xgz" podUID="af53f0dd-ff73-4a4f-aa1b-adf866956e64" containerName="authorino" containerID="cri-o://b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1" gracePeriod=30 Aug 04 10:35:11.186401 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.186378 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-q9xgz" Aug 04 10:35:11.344808 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.344775 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv487\" (UniqueName: \"kubernetes.io/projected/af53f0dd-ff73-4a4f-aa1b-adf866956e64-kube-api-access-xv487\") pod \"af53f0dd-ff73-4a4f-aa1b-adf866956e64\" (UID: \"af53f0dd-ff73-4a4f-aa1b-adf866956e64\") " Aug 04 10:35:11.346915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.346883 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af53f0dd-ff73-4a4f-aa1b-adf866956e64-kube-api-access-xv487" (OuterVolumeSpecName: "kube-api-access-xv487") pod "af53f0dd-ff73-4a4f-aa1b-adf866956e64" (UID: "af53f0dd-ff73-4a4f-aa1b-adf866956e64"). InnerVolumeSpecName "kube-api-access-xv487". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:35:11.446223 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.446198 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv487\" (UniqueName: \"kubernetes.io/projected/af53f0dd-ff73-4a4f-aa1b-adf866956e64-kube-api-access-xv487\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:35:11.944904 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.944875 2577 generic.go:356] "Generic (PLEG): container finished" podID="af53f0dd-ff73-4a4f-aa1b-adf866956e64" containerID="b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1" exitCode=0 Aug 04 10:35:11.945287 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.944920 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-q9xgz" Aug 04 10:35:11.945287 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.944925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-q9xgz" event={"ID":"af53f0dd-ff73-4a4f-aa1b-adf866956e64","Type":"ContainerDied","Data":"b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1"} Aug 04 10:35:11.945287 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.945015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-q9xgz" event={"ID":"af53f0dd-ff73-4a4f-aa1b-adf866956e64","Type":"ContainerDied","Data":"cf7699fe642f63ca2116c131725b8517a6dd1d7c63d72ff687804e55346fd814"} Aug 04 10:35:11.945287 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.945032 2577 scope.go:117] "RemoveContainer" containerID="b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1" Aug 04 10:35:11.953711 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.953692 2577 scope.go:117] "RemoveContainer" containerID="b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1" Aug 04 10:35:11.953985 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:35:11.953961 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1\": container with ID starting with b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1 not found: ID does not exist" containerID="b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1" Aug 04 10:35:11.954073 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.953992 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1"} err="failed to get container status \"b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1\": rpc error: code = NotFound desc = could not find container \"b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1\": container with ID starting with b63ee282fd0afeffe8a2fbd6644c42e577f9239d406cf1bba2a3f4255aa946c1 not found: ID does not exist" Aug 04 10:35:11.959287 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.959260 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-q9xgz"] Aug 04 10:35:11.962852 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:11.962819 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-q9xgz"] Aug 04 10:35:13.532142 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:13.532111 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af53f0dd-ff73-4a4f-aa1b-adf866956e64" path="/var/lib/kubelet/pods/af53f0dd-ff73-4a4f-aa1b-adf866956e64/volumes" Aug 04 10:35:37.043302 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.043259 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-7lcsl"] Aug 04 10:35:37.043811 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.043790 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af53f0dd-ff73-4a4f-aa1b-adf866956e64" containerName="authorino" Aug 04 10:35:37.043919 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.043814 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af53f0dd-ff73-4a4f-aa1b-adf866956e64" containerName="authorino" Aug 04 10:35:37.043971 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.043943 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af53f0dd-ff73-4a4f-aa1b-adf866956e64" containerName="authorino" Aug 04 10:35:37.046877 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.046857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" Aug 04 10:35:37.054685 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.054662 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-7lcsl"] Aug 04 10:35:37.128395 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.128359 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-7lcsl"] Aug 04 10:35:37.128637 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:35:37.128615 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gk28g], unattached volumes=[], failed to process volumes=[kube-api-access-gk28g]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" podUID="b6278ecd-41fc-43d9-8703-1d600ecd478c" Aug 04 10:35:37.150712 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.150681 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-z8s46"] Aug 04 10:35:37.151955 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.151933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk28g\" (UniqueName: \"kubernetes.io/projected/b6278ecd-41fc-43d9-8703-1d600ecd478c-kube-api-access-gk28g\") pod \"authorino-5dc7ccb5f-7lcsl\" (UID: \"b6278ecd-41fc-43d9-8703-1d600ecd478c\") " pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" Aug 04 10:35:37.152989 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.152971 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:37.155021 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.155004 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 10:35:37.161262 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.161239 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-z8s46"] Aug 04 10:35:37.252659 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.252632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7b9\" (UniqueName: \"kubernetes.io/projected/d4980798-3918-4caa-a2df-47382c4e330a-kube-api-access-vt7b9\") pod \"authorino-687bd45585-z8s46\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:37.252809 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.252675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4980798-3918-4caa-a2df-47382c4e330a-tls-cert\") pod \"authorino-687bd45585-z8s46\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:37.252809 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.252778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk28g\" (UniqueName: \"kubernetes.io/projected/b6278ecd-41fc-43d9-8703-1d600ecd478c-kube-api-access-gk28g\") pod \"authorino-5dc7ccb5f-7lcsl\" (UID: \"b6278ecd-41fc-43d9-8703-1d600ecd478c\") " pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" Aug 04 10:35:37.262312 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.262285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk28g\" (UniqueName: \"kubernetes.io/projected/b6278ecd-41fc-43d9-8703-1d600ecd478c-kube-api-access-gk28g\") pod \"authorino-5dc7ccb5f-7lcsl\" (UID: \"b6278ecd-41fc-43d9-8703-1d600ecd478c\") " pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" Aug 04 10:35:37.275005 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.274983 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-z8s46"] Aug 04 10:35:37.275216 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:35:37.275198 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vt7b9 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-z8s46" podUID="d4980798-3918-4caa-a2df-47382c4e330a" Aug 04 10:35:37.304741 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.304694 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84b7bfc4fc-7b7dt"] Aug 04 10:35:37.306901 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.306884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.316425 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.316403 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84b7bfc4fc-7b7dt"] Aug 04 10:35:37.353366 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.353337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7b9\" (UniqueName: \"kubernetes.io/projected/d4980798-3918-4caa-a2df-47382c4e330a-kube-api-access-vt7b9\") pod \"authorino-687bd45585-z8s46\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:37.353485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.353370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdght\" (UniqueName: \"kubernetes.io/projected/c7186387-f372-4102-8e7e-d499fa0f3650-kube-api-access-sdght\") pod \"authorino-84b7bfc4fc-7b7dt\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.353485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.353398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4980798-3918-4caa-a2df-47382c4e330a-tls-cert\") pod \"authorino-687bd45585-z8s46\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:37.353485 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.353466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7186387-f372-4102-8e7e-d499fa0f3650-tls-cert\") pod \"authorino-84b7bfc4fc-7b7dt\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.355688 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.355666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4980798-3918-4caa-a2df-47382c4e330a-tls-cert\") pod \"authorino-687bd45585-z8s46\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:37.362069 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.362040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7b9\" (UniqueName: \"kubernetes.io/projected/d4980798-3918-4caa-a2df-47382c4e330a-kube-api-access-vt7b9\") pod \"authorino-687bd45585-z8s46\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:37.454100 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.454072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdght\" (UniqueName: \"kubernetes.io/projected/c7186387-f372-4102-8e7e-d499fa0f3650-kube-api-access-sdght\") pod \"authorino-84b7bfc4fc-7b7dt\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.454206 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.454126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7186387-f372-4102-8e7e-d499fa0f3650-tls-cert\") pod \"authorino-84b7bfc4fc-7b7dt\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.456552 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.456526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7186387-f372-4102-8e7e-d499fa0f3650-tls-cert\") pod \"authorino-84b7bfc4fc-7b7dt\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.462676 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.462656 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdght\" (UniqueName: \"kubernetes.io/projected/c7186387-f372-4102-8e7e-d499fa0f3650-kube-api-access-sdght\") pod \"authorino-84b7bfc4fc-7b7dt\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.616875 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.616809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:35:37.748907 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:37.748877 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84b7bfc4fc-7b7dt"] Aug 04 10:35:37.750761 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:35:37.750736 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7186387_f372_4102_8e7e_d499fa0f3650.slice/crio-6fae9de9d1c1e2d3d5c5485505428a2656733ff89dbb27def7be4cb1bb5d1fbf WatchSource:0}: Error finding container 6fae9de9d1c1e2d3d5c5485505428a2656733ff89dbb27def7be4cb1bb5d1fbf: Status 404 returned error can't find the container with id 6fae9de9d1c1e2d3d5c5485505428a2656733ff89dbb27def7be4cb1bb5d1fbf Aug 04 10:35:38.089512 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.089479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" event={"ID":"c7186387-f372-4102-8e7e-d499fa0f3650","Type":"ContainerStarted","Data":"6fae9de9d1c1e2d3d5c5485505428a2656733ff89dbb27def7be4cb1bb5d1fbf"} Aug 04 10:35:38.089942 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.089570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" Aug 04 10:35:38.089942 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.089572 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:38.096206 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.096187 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" Aug 04 10:35:38.099000 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.098978 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:38.160595 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.160575 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt7b9\" (UniqueName: \"kubernetes.io/projected/d4980798-3918-4caa-a2df-47382c4e330a-kube-api-access-vt7b9\") pod \"d4980798-3918-4caa-a2df-47382c4e330a\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " Aug 04 10:35:38.160711 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.160611 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4980798-3918-4caa-a2df-47382c4e330a-tls-cert\") pod \"d4980798-3918-4caa-a2df-47382c4e330a\" (UID: \"d4980798-3918-4caa-a2df-47382c4e330a\") " Aug 04 10:35:38.160711 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.160643 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk28g\" (UniqueName: \"kubernetes.io/projected/b6278ecd-41fc-43d9-8703-1d600ecd478c-kube-api-access-gk28g\") pod \"b6278ecd-41fc-43d9-8703-1d600ecd478c\" (UID: \"b6278ecd-41fc-43d9-8703-1d600ecd478c\") " Aug 04 10:35:38.162720 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.162693 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4980798-3918-4caa-a2df-47382c4e330a-kube-api-access-vt7b9" (OuterVolumeSpecName: "kube-api-access-vt7b9") pod "d4980798-3918-4caa-a2df-47382c4e330a" (UID: "d4980798-3918-4caa-a2df-47382c4e330a"). InnerVolumeSpecName "kube-api-access-vt7b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:35:38.162801 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.162741 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4980798-3918-4caa-a2df-47382c4e330a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d4980798-3918-4caa-a2df-47382c4e330a" (UID: "d4980798-3918-4caa-a2df-47382c4e330a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:35:38.162801 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.162751 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6278ecd-41fc-43d9-8703-1d600ecd478c-kube-api-access-gk28g" (OuterVolumeSpecName: "kube-api-access-gk28g") pod "b6278ecd-41fc-43d9-8703-1d600ecd478c" (UID: "b6278ecd-41fc-43d9-8703-1d600ecd478c"). InnerVolumeSpecName "kube-api-access-gk28g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:35:38.261365 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.261343 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4980798-3918-4caa-a2df-47382c4e330a-tls-cert\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:35:38.261365 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.261364 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk28g\" (UniqueName: \"kubernetes.io/projected/b6278ecd-41fc-43d9-8703-1d600ecd478c-kube-api-access-gk28g\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:35:38.261480 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:38.261375 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt7b9\" (UniqueName: \"kubernetes.io/projected/d4980798-3918-4caa-a2df-47382c4e330a-kube-api-access-vt7b9\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:35:39.096365 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.096333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-z8s46" Aug 04 10:35:39.096820 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.096330 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" event={"ID":"c7186387-f372-4102-8e7e-d499fa0f3650","Type":"ContainerStarted","Data":"78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b"} Aug 04 10:35:39.096820 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.096521 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-7lcsl" Aug 04 10:35:39.119471 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.119112 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" podStartSLOduration=1.39777094 podStartE2EDuration="2.119097592s" totalImagesPullingTime="721.326652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:35:39.118888543 +0000 UTC m=+500.070296094" watchObservedRunningTime="2026-08-04 10:35:39.119097592 +0000 UTC m=+500.070505143" Aug 04 10:35:39.146984 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.146955 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-z8s46"] Aug 04 10:35:39.156717 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.156692 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-z8s46"] Aug 04 10:35:39.160152 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.160117 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lvpdb"] Aug 04 10:35:39.160343 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.160320 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" podUID="982a7c26-05e2-4a8c-9432-c5f4cd89463a" containerName="authorino" containerID="cri-o://3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28" gracePeriod=30 Aug 04 10:35:39.178620 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.178597 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-7lcsl"] Aug 04 10:35:39.180587 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.180565 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-7lcsl"] Aug 04 10:35:39.405756 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.405733 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" Aug 04 10:35:39.469697 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.469670 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r98g\" (UniqueName: \"kubernetes.io/projected/982a7c26-05e2-4a8c-9432-c5f4cd89463a-kube-api-access-9r98g\") pod \"982a7c26-05e2-4a8c-9432-c5f4cd89463a\" (UID: \"982a7c26-05e2-4a8c-9432-c5f4cd89463a\") " Aug 04 10:35:39.471657 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.471633 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982a7c26-05e2-4a8c-9432-c5f4cd89463a-kube-api-access-9r98g" (OuterVolumeSpecName: "kube-api-access-9r98g") pod "982a7c26-05e2-4a8c-9432-c5f4cd89463a" (UID: "982a7c26-05e2-4a8c-9432-c5f4cd89463a"). InnerVolumeSpecName "kube-api-access-9r98g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:35:39.532790 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.532764 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6278ecd-41fc-43d9-8703-1d600ecd478c" path="/var/lib/kubelet/pods/b6278ecd-41fc-43d9-8703-1d600ecd478c/volumes" Aug 04 10:35:39.533050 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.533036 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4980798-3918-4caa-a2df-47382c4e330a" path="/var/lib/kubelet/pods/d4980798-3918-4caa-a2df-47382c4e330a/volumes" Aug 04 10:35:39.570352 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:39.570325 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r98g\" (UniqueName: \"kubernetes.io/projected/982a7c26-05e2-4a8c-9432-c5f4cd89463a-kube-api-access-9r98g\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:35:40.102556 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.102521 2577 generic.go:356] "Generic (PLEG): container finished" podID="982a7c26-05e2-4a8c-9432-c5f4cd89463a" containerID="3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28" exitCode=0 Aug 04 10:35:40.102997 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.102563 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" Aug 04 10:35:40.102997 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.102617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" event={"ID":"982a7c26-05e2-4a8c-9432-c5f4cd89463a","Type":"ContainerDied","Data":"3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28"} Aug 04 10:35:40.102997 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.102659 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-lvpdb" event={"ID":"982a7c26-05e2-4a8c-9432-c5f4cd89463a","Type":"ContainerDied","Data":"ff825ccdcf2b041e7b2c5ad5b9853278ffbb8c42a45161ee37fb28b986855296"} Aug 04 10:35:40.102997 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.102678 2577 scope.go:117] "RemoveContainer" containerID="3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28" Aug 04 10:35:40.111588 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.111571 2577 scope.go:117] "RemoveContainer" containerID="3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28" Aug 04 10:35:40.111871 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:35:40.111848 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28\": container with ID starting with 3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28 not found: ID does not exist" containerID="3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28" Aug 04 10:35:40.111938 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.111882 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28"} err="failed to get container status \"3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28\": rpc error: code = NotFound desc = could not find container \"3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28\": container with ID starting with 3da330d3369f25ac3c9f55f2b19471af813a76e50404701e763faf1200960e28 not found: ID does not exist" Aug 04 10:35:40.120010 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.119986 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lvpdb"] Aug 04 10:35:40.124400 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:40.124375 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-lvpdb"] Aug 04 10:35:41.531844 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:35:41.531806 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="982a7c26-05e2-4a8c-9432-c5f4cd89463a" path="/var/lib/kubelet/pods/982a7c26-05e2-4a8c-9432-c5f4cd89463a/volumes" Aug 04 10:36:19.558467 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.558432 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q"] Aug 04 10:36:19.558798 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.558778 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982a7c26-05e2-4a8c-9432-c5f4cd89463a" containerName="authorino" Aug 04 10:36:19.558798 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.558788 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="982a7c26-05e2-4a8c-9432-c5f4cd89463a" containerName="authorino" Aug 04 10:36:19.558892 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.558872 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="982a7c26-05e2-4a8c-9432-c5f4cd89463a" containerName="authorino" Aug 04 10:36:19.562158 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.562143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.564175 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.564155 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 10:36:19.564537 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.564521 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Aug 04 10:36:19.564735 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.564721 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6qxht\"" Aug 04 10:36:19.564794 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.564741 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 10:36:19.577378 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.577346 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q"] Aug 04 10:36:19.693935 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.693902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-maas-api-tls\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.694132 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.694108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd94n\" (UniqueName: \"kubernetes.io/projected/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-kube-api-access-kd94n\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.694268 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.694145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-openshift-service-ca\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.794827 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.794757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd94n\" (UniqueName: \"kubernetes.io/projected/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-kube-api-access-kd94n\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.794827 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.794789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-openshift-service-ca\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.794827 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.794807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-maas-api-tls\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.795537 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.795514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-openshift-service-ca\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.797199 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.797176 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-maas-api-tls\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.803564 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.803548 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd94n\" (UniqueName: \"kubernetes.io/projected/fe182b94-9c7d-4a3c-bde7-8fcae9ff95af-kube-api-access-kd94n\") pod \"maas-api-7bc9f979dd-hlj6q\" (UID: \"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af\") " pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:19.879607 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:19.879579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:20.045266 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:20.045238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q"] Aug 04 10:36:20.047723 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:36:20.047693 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe182b94_9c7d_4a3c_bde7_8fcae9ff95af.slice/crio-c11863aaf5c0722218a3b6d2408da8236294a9d681e32f847af8ea81d7adc506 WatchSource:0}: Error finding container c11863aaf5c0722218a3b6d2408da8236294a9d681e32f847af8ea81d7adc506: Status 404 returned error can't find the container with id c11863aaf5c0722218a3b6d2408da8236294a9d681e32f847af8ea81d7adc506 Aug 04 10:36:20.318330 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:20.318255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" event={"ID":"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af","Type":"ContainerStarted","Data":"c11863aaf5c0722218a3b6d2408da8236294a9d681e32f847af8ea81d7adc506"} Aug 04 10:36:24.344705 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:24.344663 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" event={"ID":"fe182b94-9c7d-4a3c-bde7-8fcae9ff95af","Type":"ContainerStarted","Data":"a7a0a58abf91b5bbc08edbac4165b64bcae6616055ad8ed48650c139593ed6a2"} Aug 04 10:36:24.345109 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:24.344781 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:36:24.362332 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:24.362288 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" podStartSLOduration=1.451129929 podStartE2EDuration="5.362275338s" totalImagesPullingTime="3.911145409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:36:24.361216795 +0000 UTC m=+545.312624346" watchObservedRunningTime="2026-08-04 10:36:24.362275338 +0000 UTC m=+545.313682886" Aug 04 10:36:30.354533 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:36:30.354503 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7bc9f979dd-hlj6q" Aug 04 10:37:08.148192 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.148160 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s"] Aug 04 10:37:08.151702 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.151683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.154279 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.154250 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 10:37:08.154393 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.154254 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 10:37:08.154393 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.154253 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Aug 04 10:37:08.154393 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.154254 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ps8jd\"" Aug 04 10:37:08.163035 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.163011 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s"] Aug 04 10:37:08.314298 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.314267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz6d2\" (UniqueName: \"kubernetes.io/projected/afd6430d-bd8d-4b7b-8dbc-286e3a173095-kube-api-access-mz6d2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.314298 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.314300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.314456 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.314332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.314456 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.314420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.314526 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.314476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.314526 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.314502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd6430d-bd8d-4b7b-8dbc-286e3a173095-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.414985 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.414922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.414985 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.414960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd6430d-bd8d-4b7b-8dbc-286e3a173095-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.415118 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.414994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6d2\" (UniqueName: \"kubernetes.io/projected/afd6430d-bd8d-4b7b-8dbc-286e3a173095-kube-api-access-mz6d2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.415118 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.415020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.415118 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.415068 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.415243 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.415120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.415456 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.415437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.415530 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.415470 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.415530 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.415508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.417199 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.417179 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afd6430d-bd8d-4b7b-8dbc-286e3a173095-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.417599 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.417579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afd6430d-bd8d-4b7b-8dbc-286e3a173095-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.422107 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.422085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6d2\" (UniqueName: \"kubernetes.io/projected/afd6430d-bd8d-4b7b-8dbc-286e3a173095-kube-api-access-mz6d2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ff7s\" (UID: \"afd6430d-bd8d-4b7b-8dbc-286e3a173095\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.462848 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.462813 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:08.598363 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:08.598339 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s"] Aug 04 10:37:08.601350 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:37:08.601325 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd6430d_bd8d_4b7b_8dbc_286e3a173095.slice/crio-a94e7e8b479b8496b4542af51f517df34438dff52b0495fcf874f0959c052717 WatchSource:0}: Error finding container a94e7e8b479b8496b4542af51f517df34438dff52b0495fcf874f0959c052717: Status 404 returned error can't find the container with id a94e7e8b479b8496b4542af51f517df34438dff52b0495fcf874f0959c052717 Aug 04 10:37:09.588577 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:09.588533 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" event={"ID":"afd6430d-bd8d-4b7b-8dbc-286e3a173095","Type":"ContainerStarted","Data":"a94e7e8b479b8496b4542af51f517df34438dff52b0495fcf874f0959c052717"} Aug 04 10:37:10.595481 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:10.595443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" event={"ID":"afd6430d-bd8d-4b7b-8dbc-286e3a173095","Type":"ContainerStarted","Data":"20231a2b429d4d32c7e065daeeffc8d1be78d0146a3cab80c203ac4651e05450"} Aug 04 10:37:10.612077 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:10.612026 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" podStartSLOduration=1.388856385 podStartE2EDuration="2.612008228s" totalImagesPullingTime="1.223151843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:10.611248496 +0000 UTC m=+591.562656047" watchObservedRunningTime="2026-08-04 10:37:10.612008228 +0000 UTC m=+591.563415782" Aug 04 10:37:18.463121 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:18.463077 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:18.463509 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:18.463148 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:18.475676 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:18.475647 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:18.648132 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:18.648097 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ff7s" Aug 04 10:37:19.524980 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:19.524946 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:37:19.527394 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:19.527367 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:37:47.742643 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.742563 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p"] Aug 04 10:37:47.746104 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.746088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.748140 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.748118 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 10:37:47.757550 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.757527 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p"] Aug 04 10:37:47.875052 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.875015 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.875052 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.875055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c1aa1a-21d9-465d-8225-1ee260a7d299-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.875257 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.875079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.875257 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.875108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.875257 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.875185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cclk\" (UniqueName: \"kubernetes.io/projected/c2c1aa1a-21d9-465d-8225-1ee260a7d299-kube-api-access-9cclk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.875257 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.875218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976200 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976200 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c1aa1a-21d9-465d-8225-1ee260a7d299-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976445 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976445 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976445 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cclk\" (UniqueName: \"kubernetes.io/projected/c2c1aa1a-21d9-465d-8225-1ee260a7d299-kube-api-access-9cclk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976445 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976703 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976678 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976788 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.976788 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.976757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.978583 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.978555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2c1aa1a-21d9-465d-8225-1ee260a7d299-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.978751 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.978736 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2c1aa1a-21d9-465d-8225-1ee260a7d299-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:47.984305 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:47.984281 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cclk\" (UniqueName: \"kubernetes.io/projected/c2c1aa1a-21d9-465d-8225-1ee260a7d299-kube-api-access-9cclk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vs77p\" (UID: \"c2c1aa1a-21d9-465d-8225-1ee260a7d299\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:48.056155 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:48.056127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:48.189893 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:48.189857 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p"] Aug 04 10:37:48.192370 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:37:48.192328 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2c1aa1a_21d9_465d_8225_1ee260a7d299.slice/crio-268808fb8352b3e5bfd38d3775a6712b0b3002722bd5184e5f7dac1df17d4c68 WatchSource:0}: Error finding container 268808fb8352b3e5bfd38d3775a6712b0b3002722bd5184e5f7dac1df17d4c68: Status 404 returned error can't find the container with id 268808fb8352b3e5bfd38d3775a6712b0b3002722bd5184e5f7dac1df17d4c68 Aug 04 10:37:48.193708 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:48.193692 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:37:48.802288 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:48.802247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" event={"ID":"c2c1aa1a-21d9-465d-8225-1ee260a7d299","Type":"ContainerStarted","Data":"c24c570624b07e9406ece2a4f974486a4a2ccf35a9d1fb5340853d34f652fd99"} Aug 04 10:37:48.802657 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:48.802297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" event={"ID":"c2c1aa1a-21d9-465d-8225-1ee260a7d299","Type":"ContainerStarted","Data":"268808fb8352b3e5bfd38d3775a6712b0b3002722bd5184e5f7dac1df17d4c68"} Aug 04 10:37:48.826162 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:48.826110 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" podStartSLOduration=1.6546565869999998 podStartE2EDuration="1.826093582s" totalImagesPullingTime="171.436995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:37:48.824867848 +0000 UTC m=+629.776275393" watchObservedRunningTime="2026-08-04 10:37:48.826093582 +0000 UTC m=+629.777501132" Aug 04 10:37:58.056471 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:58.056436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:58.056877 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:58.056491 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:58.068729 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:58.068703 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:37:58.870229 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:37:58.870200 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vs77p" Aug 04 10:38:54.725211 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.725176 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64798b4f48-cvjqv"] Aug 04 10:38:54.728736 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.728718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:54.737947 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.737919 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64798b4f48-cvjqv"] Aug 04 10:38:54.763491 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.763462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cp4\" (UniqueName: \"kubernetes.io/projected/5cf45ed7-efd2-499b-896d-b237805ae0cd-kube-api-access-j4cp4\") pod \"authorino-64798b4f48-cvjqv\" (UID: \"5cf45ed7-efd2-499b-896d-b237805ae0cd\") " pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:54.763597 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.763567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf45ed7-efd2-499b-896d-b237805ae0cd-tls-cert\") pod \"authorino-64798b4f48-cvjqv\" (UID: \"5cf45ed7-efd2-499b-896d-b237805ae0cd\") " pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:54.864789 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.864760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf45ed7-efd2-499b-896d-b237805ae0cd-tls-cert\") pod \"authorino-64798b4f48-cvjqv\" (UID: \"5cf45ed7-efd2-499b-896d-b237805ae0cd\") " pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:54.864920 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.864821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cp4\" (UniqueName: \"kubernetes.io/projected/5cf45ed7-efd2-499b-896d-b237805ae0cd-kube-api-access-j4cp4\") pod \"authorino-64798b4f48-cvjqv\" (UID: \"5cf45ed7-efd2-499b-896d-b237805ae0cd\") " pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:54.867296 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.867266 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5cf45ed7-efd2-499b-896d-b237805ae0cd-tls-cert\") pod \"authorino-64798b4f48-cvjqv\" (UID: \"5cf45ed7-efd2-499b-896d-b237805ae0cd\") " pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:54.873438 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:54.873413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cp4\" (UniqueName: \"kubernetes.io/projected/5cf45ed7-efd2-499b-896d-b237805ae0cd-kube-api-access-j4cp4\") pod \"authorino-64798b4f48-cvjqv\" (UID: \"5cf45ed7-efd2-499b-896d-b237805ae0cd\") " pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:55.038573 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:55.038548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64798b4f48-cvjqv" Aug 04 10:38:55.172844 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:55.172806 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64798b4f48-cvjqv"] Aug 04 10:38:55.175504 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:38:55.175473 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cf45ed7_efd2_499b_896d_b237805ae0cd.slice/crio-5c210a594a1384169eebcbeaf113eb6e72a26fc5d11c95c282d9fcf226ce53be WatchSource:0}: Error finding container 5c210a594a1384169eebcbeaf113eb6e72a26fc5d11c95c282d9fcf226ce53be: Status 404 returned error can't find the container with id 5c210a594a1384169eebcbeaf113eb6e72a26fc5d11c95c282d9fcf226ce53be Aug 04 10:38:56.170676 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.170587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64798b4f48-cvjqv" event={"ID":"5cf45ed7-efd2-499b-896d-b237805ae0cd","Type":"ContainerStarted","Data":"828a3a6cb36da873818a2cf07d1e52ee627c84ba478c2957199e5774c5275354"} Aug 04 10:38:56.170676 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.170646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64798b4f48-cvjqv" event={"ID":"5cf45ed7-efd2-499b-896d-b237805ae0cd","Type":"ContainerStarted","Data":"5c210a594a1384169eebcbeaf113eb6e72a26fc5d11c95c282d9fcf226ce53be"} Aug 04 10:38:56.190491 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.190429 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64798b4f48-cvjqv" podStartSLOduration=1.4887667900000001 podStartE2EDuration="2.190406688s" totalImagesPullingTime="701.639898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:38:56.189489679 +0000 UTC m=+697.140897230" watchObservedRunningTime="2026-08-04 10:38:56.190406688 +0000 UTC m=+697.141814241" Aug 04 10:38:56.227897 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.227856 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84b7bfc4fc-7b7dt"] Aug 04 10:38:56.228340 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.228307 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" podUID="c7186387-f372-4102-8e7e-d499fa0f3650" containerName="authorino" containerID="cri-o://78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b" gracePeriod=30 Aug 04 10:38:56.485737 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.485711 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:38:56.581069 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.581034 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdght\" (UniqueName: \"kubernetes.io/projected/c7186387-f372-4102-8e7e-d499fa0f3650-kube-api-access-sdght\") pod \"c7186387-f372-4102-8e7e-d499fa0f3650\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " Aug 04 10:38:56.581247 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.581087 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7186387-f372-4102-8e7e-d499fa0f3650-tls-cert\") pod \"c7186387-f372-4102-8e7e-d499fa0f3650\" (UID: \"c7186387-f372-4102-8e7e-d499fa0f3650\") " Aug 04 10:38:56.583224 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.583193 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7186387-f372-4102-8e7e-d499fa0f3650-kube-api-access-sdght" (OuterVolumeSpecName: "kube-api-access-sdght") pod "c7186387-f372-4102-8e7e-d499fa0f3650" (UID: "c7186387-f372-4102-8e7e-d499fa0f3650"). InnerVolumeSpecName "kube-api-access-sdght". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:38:56.592156 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.592125 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7186387-f372-4102-8e7e-d499fa0f3650-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c7186387-f372-4102-8e7e-d499fa0f3650" (UID: "c7186387-f372-4102-8e7e-d499fa0f3650"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:38:56.682353 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.682312 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdght\" (UniqueName: \"kubernetes.io/projected/c7186387-f372-4102-8e7e-d499fa0f3650-kube-api-access-sdght\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:38:56.682353 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:56.682354 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7186387-f372-4102-8e7e-d499fa0f3650-tls-cert\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:38:57.177761 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.177680 2577 generic.go:356] "Generic (PLEG): container finished" podID="c7186387-f372-4102-8e7e-d499fa0f3650" containerID="78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b" exitCode=0 Aug 04 10:38:57.177761 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.177737 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" Aug 04 10:38:57.178280 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.177772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" event={"ID":"c7186387-f372-4102-8e7e-d499fa0f3650","Type":"ContainerDied","Data":"78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b"} Aug 04 10:38:57.178280 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.177822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84b7bfc4fc-7b7dt" event={"ID":"c7186387-f372-4102-8e7e-d499fa0f3650","Type":"ContainerDied","Data":"6fae9de9d1c1e2d3d5c5485505428a2656733ff89dbb27def7be4cb1bb5d1fbf"} Aug 04 10:38:57.178280 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.177855 2577 scope.go:117] "RemoveContainer" containerID="78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b" Aug 04 10:38:57.188417 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.188398 2577 scope.go:117] "RemoveContainer" containerID="78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b" Aug 04 10:38:57.188694 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:38:57.188671 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b\": container with ID starting with 78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b not found: ID does not exist" containerID="78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b" Aug 04 10:38:57.188782 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.188706 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b"} err="failed to get container status \"78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b\": rpc error: code = NotFound desc = could not find container \"78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b\": container with ID starting with 78b0f0d2c44a746d01789d2344daf89d39104454bd2e36a99964a51a0534b45b not found: ID does not exist" Aug 04 10:38:57.202427 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.202395 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84b7bfc4fc-7b7dt"] Aug 04 10:38:57.210674 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.210650 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84b7bfc4fc-7b7dt"] Aug 04 10:38:57.532818 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:38:57.532784 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7186387-f372-4102-8e7e-d499fa0f3650" path="/var/lib/kubelet/pods/c7186387-f372-4102-8e7e-d499fa0f3650/volumes" Aug 04 10:42:19.564956 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:42:19.564925 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:42:19.565385 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:42:19.565351 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:47:19.611915 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:47:19.611883 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:47:19.615940 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:47:19.615914 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:52:19.649529 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:52:19.649504 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:52:19.656484 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:52:19.656462 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:56:52.363002 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.362963 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq"] Aug 04 10:56:52.363616 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.363598 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7186387-f372-4102-8e7e-d499fa0f3650" containerName="authorino" Aug 04 10:56:52.363662 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.363618 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7186387-f372-4102-8e7e-d499fa0f3650" containerName="authorino" Aug 04 10:56:52.363736 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.363726 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7186387-f372-4102-8e7e-d499fa0f3650" containerName="authorino" Aug 04 10:56:52.367070 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.367054 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.369247 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.369227 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5f7d6dc8-serving-cert\"" Aug 04 10:56:52.376260 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.376241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq"] Aug 04 10:56:52.498469 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.498447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18088bb-61c0-42bb-899e-df250e0271dc-maas-api-tls\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.498584 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.498512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18088bb-61c0-42bb-899e-df250e0271dc-openshift-service-ca\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.498584 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.498538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdf52\" (UniqueName: \"kubernetes.io/projected/a18088bb-61c0-42bb-899e-df250e0271dc-kube-api-access-sdf52\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.599672 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.599646 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18088bb-61c0-42bb-899e-df250e0271dc-openshift-service-ca\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.599785 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.599678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdf52\" (UniqueName: \"kubernetes.io/projected/a18088bb-61c0-42bb-899e-df250e0271dc-kube-api-access-sdf52\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.599785 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.599702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18088bb-61c0-42bb-899e-df250e0271dc-maas-api-tls\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.600317 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.600295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18088bb-61c0-42bb-899e-df250e0271dc-openshift-service-ca\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.602111 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.602086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18088bb-61c0-42bb-899e-df250e0271dc-maas-api-tls\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.607523 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.607498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdf52\" (UniqueName: \"kubernetes.io/projected/a18088bb-61c0-42bb-899e-df250e0271dc-kube-api-access-sdf52\") pod \"maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.678852 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.678789 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:52.810909 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.810873 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq"] Aug 04 10:56:52.812184 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:56:52.812158 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18088bb_61c0_42bb_899e_df250e0271dc.slice/crio-2726ca55f146c82f35feeefd3c8b736536a262c4adc31c8b5643844ecb8bdf49 WatchSource:0}: Error finding container 2726ca55f146c82f35feeefd3c8b736536a262c4adc31c8b5643844ecb8bdf49: Status 404 returned error can't find the container with id 2726ca55f146c82f35feeefd3c8b736536a262c4adc31c8b5643844ecb8bdf49 Aug 04 10:56:52.813506 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:52.813489 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 10:56:53.037723 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:53.037683 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" event={"ID":"a18088bb-61c0-42bb-899e-df250e0271dc","Type":"ContainerStarted","Data":"2726ca55f146c82f35feeefd3c8b736536a262c4adc31c8b5643844ecb8bdf49"} Aug 04 10:56:54.044936 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:54.044898 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" event={"ID":"a18088bb-61c0-42bb-899e-df250e0271dc","Type":"ContainerStarted","Data":"b4bdb3787d0e1f90791208594e1edab429fd703f56cb245231ea09b38a5bab70"} Aug 04 10:56:54.045365 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:54.044954 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:56:54.064819 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:56:54.064775 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" podStartSLOduration=1.276099593 podStartE2EDuration="2.064761466s" totalImagesPullingTime="788.661873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:56:54.062587495 +0000 UTC m=+1775.013995044" watchObservedRunningTime="2026-08-04 10:56:54.064761466 +0000 UTC m=+1775.016169015" Aug 04 10:57:00.056047 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:00.056018 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:57:16.872989 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:16.872957 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw"] Aug 04 10:57:16.876675 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:16.876658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" Aug 04 10:57:16.878711 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:16.878690 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 10:57:16.885762 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:16.885732 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw"] Aug 04 10:57:17.023669 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:17.023637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw\" (UID: \"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" Aug 04 10:57:17.124804 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:17.124711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw\" (UID: \"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" Aug 04 10:57:17.125366 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:17.125343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw\" (UID: \"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" Aug 04 10:57:17.187775 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:17.187748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" Aug 04 10:57:17.320775 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:17.320743 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw"] Aug 04 10:57:17.323093 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:57:17.323069 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13a4af2_2c0d_4e96_ae9e_f0b9fec00fb5.slice/crio-7257f038c65f7f0967b91503a208057247c068bef0eac5f8d321eefffcb61fa1 WatchSource:0}: Error finding container 7257f038c65f7f0967b91503a208057247c068bef0eac5f8d321eefffcb61fa1: Status 404 returned error can't find the container with id 7257f038c65f7f0967b91503a208057247c068bef0eac5f8d321eefffcb61fa1 Aug 04 10:57:18.182983 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:18.182908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" event={"ID":"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5","Type":"ContainerStarted","Data":"7257f038c65f7f0967b91503a208057247c068bef0eac5f8d321eefffcb61fa1"} Aug 04 10:57:19.687707 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:19.687684 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:57:19.694929 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:19.694907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 10:57:20.197864 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:20.197764 2577 generic.go:356] "Generic (PLEG): container finished" podID="a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5" containerID="e744746a7a7a3068609b945e5b44128659f1df622134d22733e2015c017b7785" exitCode=0 Aug 04 10:57:20.198045 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:20.197859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" event={"ID":"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5","Type":"ContainerDied","Data":"e744746a7a7a3068609b945e5b44128659f1df622134d22733e2015c017b7785"} Aug 04 10:57:21.338470 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:21.338446 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" Aug 04 10:57:21.363090 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:21.363065 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5-maas-api-service-ca\") pod \"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5\" (UID: \"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5\") " Aug 04 10:57:21.363406 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:21.363367 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5" (UID: "a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:57:21.464791 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:21.464715 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:22.210620 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:22.210592 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" Aug 04 10:57:22.210793 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:22.210597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw" event={"ID":"a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5","Type":"ContainerDied","Data":"7257f038c65f7f0967b91503a208057247c068bef0eac5f8d321eefffcb61fa1"} Aug 04 10:57:22.210793 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:22.210712 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7257f038c65f7f0967b91503a208057247c068bef0eac5f8d321eefffcb61fa1" Aug 04 10:57:23.074488 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.074459 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq"] Aug 04 10:57:23.074907 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.074672 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" podUID="a18088bb-61c0-42bb-899e-df250e0271dc" containerName="maas-api" containerID="cri-o://b4bdb3787d0e1f90791208594e1edab429fd703f56cb245231ea09b38a5bab70" gracePeriod=30 Aug 04 10:57:23.218409 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.218369 2577 generic.go:356] "Generic (PLEG): container finished" podID="a18088bb-61c0-42bb-899e-df250e0271dc" containerID="b4bdb3787d0e1f90791208594e1edab429fd703f56cb245231ea09b38a5bab70" exitCode=0 Aug 04 10:57:23.218584 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.218440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" event={"ID":"a18088bb-61c0-42bb-899e-df250e0271dc","Type":"ContainerDied","Data":"b4bdb3787d0e1f90791208594e1edab429fd703f56cb245231ea09b38a5bab70"} Aug 04 10:57:23.295181 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.295150 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw"] Aug 04 10:57:23.298960 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.298937 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f7d6dc8-mjlhw"] Aug 04 10:57:23.328552 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.328501 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:57:23.383874 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.383845 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdf52\" (UniqueName: \"kubernetes.io/projected/a18088bb-61c0-42bb-899e-df250e0271dc-kube-api-access-sdf52\") pod \"a18088bb-61c0-42bb-899e-df250e0271dc\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " Aug 04 10:57:23.384006 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.383886 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18088bb-61c0-42bb-899e-df250e0271dc-openshift-service-ca\") pod \"a18088bb-61c0-42bb-899e-df250e0271dc\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " Aug 04 10:57:23.384006 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.383957 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18088bb-61c0-42bb-899e-df250e0271dc-maas-api-tls\") pod \"a18088bb-61c0-42bb-899e-df250e0271dc\" (UID: \"a18088bb-61c0-42bb-899e-df250e0271dc\") " Aug 04 10:57:23.384283 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.384249 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18088bb-61c0-42bb-899e-df250e0271dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a18088bb-61c0-42bb-899e-df250e0271dc" (UID: "a18088bb-61c0-42bb-899e-df250e0271dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:57:23.386124 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.386096 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18088bb-61c0-42bb-899e-df250e0271dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a18088bb-61c0-42bb-899e-df250e0271dc" (UID: "a18088bb-61c0-42bb-899e-df250e0271dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:57:23.386124 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.386111 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18088bb-61c0-42bb-899e-df250e0271dc-kube-api-access-sdf52" (OuterVolumeSpecName: "kube-api-access-sdf52") pod "a18088bb-61c0-42bb-899e-df250e0271dc" (UID: "a18088bb-61c0-42bb-899e-df250e0271dc"). InnerVolumeSpecName "kube-api-access-sdf52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:57:23.485191 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.485159 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdf52\" (UniqueName: \"kubernetes.io/projected/a18088bb-61c0-42bb-899e-df250e0271dc-kube-api-access-sdf52\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:23.485329 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.485201 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a18088bb-61c0-42bb-899e-df250e0271dc-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:23.485329 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.485212 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a18088bb-61c0-42bb-899e-df250e0271dc-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:23.532976 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:23.532944 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5" path="/var/lib/kubelet/pods/a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5/volumes" Aug 04 10:57:24.226003 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:24.225973 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" Aug 04 10:57:24.226472 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:24.225999 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq" event={"ID":"a18088bb-61c0-42bb-899e-df250e0271dc","Type":"ContainerDied","Data":"2726ca55f146c82f35feeefd3c8b736536a262c4adc31c8b5643844ecb8bdf49"} Aug 04 10:57:24.226472 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:24.226074 2577 scope.go:117] "RemoveContainer" containerID="b4bdb3787d0e1f90791208594e1edab429fd703f56cb245231ea09b38a5bab70" Aug 04 10:57:24.243789 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:24.243767 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq"] Aug 04 10:57:24.248715 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:24.248693 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5f7d6dc8-6bf4b85c8-vtmtq"] Aug 04 10:57:25.532521 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:25.532490 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18088bb-61c0-42bb-899e-df250e0271dc" path="/var/lib/kubelet/pods/a18088bb-61c0-42bb-899e-df250e0271dc/volumes" Aug 04 10:57:26.663995 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.663961 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx"] Aug 04 10:57:26.664559 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.664536 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5" containerName="revoke-keys" Aug 04 10:57:26.664661 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.664560 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5" containerName="revoke-keys" Aug 04 10:57:26.664661 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.664593 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18088bb-61c0-42bb-899e-df250e0271dc" containerName="maas-api" Aug 04 10:57:26.664661 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.664602 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18088bb-61c0-42bb-899e-df250e0271dc" containerName="maas-api" Aug 04 10:57:26.664807 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.664705 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18088bb-61c0-42bb-899e-df250e0271dc" containerName="maas-api" Aug 04 10:57:26.664807 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.664724 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13a4af2-2c0d-4e96-ae9e-f0b9fec00fb5" containerName="revoke-keys" Aug 04 10:57:26.669112 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.669091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.675179 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.675161 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-03e71fbe-serving-cert\"" Aug 04 10:57:26.684650 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.684631 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx"] Aug 04 10:57:26.716899 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.716873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtnfz\" (UniqueName: \"kubernetes.io/projected/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-kube-api-access-qtnfz\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.717015 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.716908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-maas-api-tls\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.717116 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.717096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-openshift-service-ca\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.817637 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.817608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnfz\" (UniqueName: \"kubernetes.io/projected/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-kube-api-access-qtnfz\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.817637 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.817644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-maas-api-tls\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.817909 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.817734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-openshift-service-ca\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.818328 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.818305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-openshift-service-ca\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.820177 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.820154 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-maas-api-tls\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.825311 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.825282 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnfz\" (UniqueName: \"kubernetes.io/projected/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-kube-api-access-qtnfz\") pod \"maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:26.979078 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:26.979012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:27.141430 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:27.141398 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx"] Aug 04 10:57:27.143460 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:57:27.143437 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81268af3_7ab2_40a1_ae87_9b7c02b50c6d.slice/crio-9ee0455144887ad412030366b2e831f48c57f58b53fa5edbece9ae52829f4420 WatchSource:0}: Error finding container 9ee0455144887ad412030366b2e831f48c57f58b53fa5edbece9ae52829f4420: Status 404 returned error can't find the container with id 9ee0455144887ad412030366b2e831f48c57f58b53fa5edbece9ae52829f4420 Aug 04 10:57:27.244112 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:27.244034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" event={"ID":"81268af3-7ab2-40a1-ae87-9b7c02b50c6d","Type":"ContainerStarted","Data":"9ee0455144887ad412030366b2e831f48c57f58b53fa5edbece9ae52829f4420"} Aug 04 10:57:28.251491 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:28.251456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" event={"ID":"81268af3-7ab2-40a1-ae87-9b7c02b50c6d","Type":"ContainerStarted","Data":"aed08fd3289c67837127ef558d8268f33ce16a3d7172f037236ba9097167e186"} Aug 04 10:57:28.251919 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:28.251524 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:28.270539 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:28.270489 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" podStartSLOduration=1.9282310580000002 podStartE2EDuration="2.270471759s" totalImagesPullingTime="342.240701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:57:28.268021726 +0000 UTC m=+1809.219429276" watchObservedRunningTime="2026-08-04 10:57:28.270471759 +0000 UTC m=+1809.221879337" Aug 04 10:57:34.262300 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:34.262274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:51.603540 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.603508 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7"] Aug 04 10:57:51.611984 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.611966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" Aug 04 10:57:51.614232 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.614201 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 10:57:51.615971 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.615926 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7"] Aug 04 10:57:51.733917 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.733886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54be6521-d4fe-453b-b99f-2a0f2cf596d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7\" (UID: \"54be6521-d4fe-453b-b99f-2a0f2cf596d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" Aug 04 10:57:51.834670 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.834632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54be6521-d4fe-453b-b99f-2a0f2cf596d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7\" (UID: \"54be6521-d4fe-453b-b99f-2a0f2cf596d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" Aug 04 10:57:51.835281 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.835257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54be6521-d4fe-453b-b99f-2a0f2cf596d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7\" (UID: \"54be6521-d4fe-453b-b99f-2a0f2cf596d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" Aug 04 10:57:51.924930 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:51.924857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" Aug 04 10:57:52.057087 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:52.057060 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7"] Aug 04 10:57:52.059052 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:57:52.059019 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54be6521_d4fe_453b_b99f_2a0f2cf596d1.slice/crio-7d83dba43e975195c75ef2018ae720d8d092dd85280b5c905ca4afaee24ca1e4 WatchSource:0}: Error finding container 7d83dba43e975195c75ef2018ae720d8d092dd85280b5c905ca4afaee24ca1e4: Status 404 returned error can't find the container with id 7d83dba43e975195c75ef2018ae720d8d092dd85280b5c905ca4afaee24ca1e4 Aug 04 10:57:52.380266 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:52.380224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" event={"ID":"54be6521-d4fe-453b-b99f-2a0f2cf596d1","Type":"ContainerStarted","Data":"8218f21532550d5384ddd623efef434e39521635da1cfa6e1306b28c5221d862"} Aug 04 10:57:52.380405 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:52.380291 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" event={"ID":"54be6521-d4fe-453b-b99f-2a0f2cf596d1","Type":"ContainerStarted","Data":"7d83dba43e975195c75ef2018ae720d8d092dd85280b5c905ca4afaee24ca1e4"} Aug 04 10:57:52.397123 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:52.397075 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" podStartSLOduration=1.397058828 podStartE2EDuration="1.397058828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:57:52.395580839 +0000 UTC m=+1833.346988388" watchObservedRunningTime="2026-08-04 10:57:52.397058828 +0000 UTC m=+1833.348466377" Aug 04 10:57:53.387424 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:53.387380 2577 generic.go:356] "Generic (PLEG): container finished" podID="54be6521-d4fe-453b-b99f-2a0f2cf596d1" containerID="8218f21532550d5384ddd623efef434e39521635da1cfa6e1306b28c5221d862" exitCode=0 Aug 04 10:57:53.387851 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:53.387460 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" event={"ID":"54be6521-d4fe-453b-b99f-2a0f2cf596d1","Type":"ContainerDied","Data":"8218f21532550d5384ddd623efef434e39521635da1cfa6e1306b28c5221d862"} Aug 04 10:57:54.522222 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:54.522200 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" Aug 04 10:57:54.661196 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:54.661120 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54be6521-d4fe-453b-b99f-2a0f2cf596d1-maas-api-service-ca\") pod \"54be6521-d4fe-453b-b99f-2a0f2cf596d1\" (UID: \"54be6521-d4fe-453b-b99f-2a0f2cf596d1\") " Aug 04 10:57:54.661454 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:54.661433 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54be6521-d4fe-453b-b99f-2a0f2cf596d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54be6521-d4fe-453b-b99f-2a0f2cf596d1" (UID: "54be6521-d4fe-453b-b99f-2a0f2cf596d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:57:54.761933 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:54.761899 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54be6521-d4fe-453b-b99f-2a0f2cf596d1-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:55.400233 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:55.400201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" Aug 04 10:57:55.400406 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:55.400234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7" event={"ID":"54be6521-d4fe-453b-b99f-2a0f2cf596d1","Type":"ContainerDied","Data":"7d83dba43e975195c75ef2018ae720d8d092dd85280b5c905ca4afaee24ca1e4"} Aug 04 10:57:55.400406 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:55.400283 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d83dba43e975195c75ef2018ae720d8d092dd85280b5c905ca4afaee24ca1e4" Aug 04 10:57:58.224508 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.224470 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx"] Aug 04 10:57:58.224911 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.224733 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" podUID="81268af3-7ab2-40a1-ae87-9b7c02b50c6d" containerName="maas-api" containerID="cri-o://aed08fd3289c67837127ef558d8268f33ce16a3d7172f037236ba9097167e186" gracePeriod=30 Aug 04 10:57:58.419375 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.419337 2577 generic.go:356] "Generic (PLEG): container finished" podID="81268af3-7ab2-40a1-ae87-9b7c02b50c6d" containerID="aed08fd3289c67837127ef558d8268f33ce16a3d7172f037236ba9097167e186" exitCode=0 Aug 04 10:57:58.419541 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.419418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" event={"ID":"81268af3-7ab2-40a1-ae87-9b7c02b50c6d","Type":"ContainerDied","Data":"aed08fd3289c67837127ef558d8268f33ce16a3d7172f037236ba9097167e186"} Aug 04 10:57:58.426890 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.426861 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7"] Aug 04 10:57:58.429336 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.429315 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-03e71fbe-tknj7"] Aug 04 10:57:58.478333 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.478272 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:58.596634 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.596600 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtnfz\" (UniqueName: \"kubernetes.io/projected/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-kube-api-access-qtnfz\") pod \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " Aug 04 10:57:58.596816 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.596678 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-openshift-service-ca\") pod \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " Aug 04 10:57:58.596816 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.596711 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-maas-api-tls\") pod \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\" (UID: \"81268af3-7ab2-40a1-ae87-9b7c02b50c6d\") " Aug 04 10:57:58.597114 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.597087 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81268af3-7ab2-40a1-ae87-9b7c02b50c6d" (UID: "81268af3-7ab2-40a1-ae87-9b7c02b50c6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:57:58.598758 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.598729 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-kube-api-access-qtnfz" (OuterVolumeSpecName: "kube-api-access-qtnfz") pod "81268af3-7ab2-40a1-ae87-9b7c02b50c6d" (UID: "81268af3-7ab2-40a1-ae87-9b7c02b50c6d"). InnerVolumeSpecName "kube-api-access-qtnfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:57:58.598880 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.598755 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81268af3-7ab2-40a1-ae87-9b7c02b50c6d" (UID: "81268af3-7ab2-40a1-ae87-9b7c02b50c6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:57:58.697519 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.697480 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtnfz\" (UniqueName: \"kubernetes.io/projected/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-kube-api-access-qtnfz\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:58.697519 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.697509 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:58.697519 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:58.697523 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81268af3-7ab2-40a1-ae87-9b7c02b50c6d-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:57:59.426347 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:59.426304 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" event={"ID":"81268af3-7ab2-40a1-ae87-9b7c02b50c6d","Type":"ContainerDied","Data":"9ee0455144887ad412030366b2e831f48c57f58b53fa5edbece9ae52829f4420"} Aug 04 10:57:59.426783 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:59.426373 2577 scope.go:117] "RemoveContainer" containerID="aed08fd3289c67837127ef558d8268f33ce16a3d7172f037236ba9097167e186" Aug 04 10:57:59.426783 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:59.426316 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx" Aug 04 10:57:59.462752 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:59.462718 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx"] Aug 04 10:57:59.483116 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:59.483081 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03e71fbe-69b86c9cfc-cr4hx"] Aug 04 10:57:59.532525 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:59.532492 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54be6521-d4fe-453b-b99f-2a0f2cf596d1" path="/var/lib/kubelet/pods/54be6521-d4fe-453b-b99f-2a0f2cf596d1/volumes" Aug 04 10:57:59.532887 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:57:59.532874 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81268af3-7ab2-40a1-ae87-9b7c02b50c6d" path="/var/lib/kubelet/pods/81268af3-7ab2-40a1-ae87-9b7c02b50c6d/volumes" Aug 04 10:58:02.733737 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.733698 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p"] Aug 04 10:58:02.734285 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.734266 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81268af3-7ab2-40a1-ae87-9b7c02b50c6d" containerName="maas-api" Aug 04 10:58:02.734285 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.734286 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81268af3-7ab2-40a1-ae87-9b7c02b50c6d" containerName="maas-api" Aug 04 10:58:02.734421 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.734297 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54be6521-d4fe-453b-b99f-2a0f2cf596d1" containerName="revoke-keys" Aug 04 10:58:02.734421 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.734303 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="54be6521-d4fe-453b-b99f-2a0f2cf596d1" containerName="revoke-keys" Aug 04 10:58:02.734421 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.734393 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81268af3-7ab2-40a1-ae87-9b7c02b50c6d" containerName="maas-api" Aug 04 10:58:02.734421 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.734403 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="54be6521-d4fe-453b-b99f-2a0f2cf596d1" containerName="revoke-keys" Aug 04 10:58:02.738864 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.738843 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.740759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.740735 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fb58dd84-serving-cert\"" Aug 04 10:58:02.750619 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.750592 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p"] Aug 04 10:58:02.839239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.839188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1038c84e-5e62-425c-addd-b1110c2a6e7e-maas-api-tls\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.839239 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.839243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtvv\" (UniqueName: \"kubernetes.io/projected/1038c84e-5e62-425c-addd-b1110c2a6e7e-kube-api-access-kgtvv\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.839478 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.839343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1038c84e-5e62-425c-addd-b1110c2a6e7e-openshift-service-ca\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.940297 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.940261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1038c84e-5e62-425c-addd-b1110c2a6e7e-maas-api-tls\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.940297 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.940301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtvv\" (UniqueName: \"kubernetes.io/projected/1038c84e-5e62-425c-addd-b1110c2a6e7e-kube-api-access-kgtvv\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.940528 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.940354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1038c84e-5e62-425c-addd-b1110c2a6e7e-openshift-service-ca\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.941085 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.941062 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1038c84e-5e62-425c-addd-b1110c2a6e7e-openshift-service-ca\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.942729 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.942706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1038c84e-5e62-425c-addd-b1110c2a6e7e-maas-api-tls\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:02.948548 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:02.948528 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtvv\" (UniqueName: \"kubernetes.io/projected/1038c84e-5e62-425c-addd-b1110c2a6e7e-kube-api-access-kgtvv\") pod \"maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:03.049500 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:03.049463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:03.184609 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:03.184583 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p"] Aug 04 10:58:03.186734 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:58:03.186707 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1038c84e_5e62_425c_addd_b1110c2a6e7e.slice/crio-1e57233b7dafe9bef0008fd5d3575cd7675a6f43d4df02412836f63ede8e3a3a WatchSource:0}: Error finding container 1e57233b7dafe9bef0008fd5d3575cd7675a6f43d4df02412836f63ede8e3a3a: Status 404 returned error can't find the container with id 1e57233b7dafe9bef0008fd5d3575cd7675a6f43d4df02412836f63ede8e3a3a Aug 04 10:58:03.451220 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:03.451136 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" event={"ID":"1038c84e-5e62-425c-addd-b1110c2a6e7e","Type":"ContainerStarted","Data":"1e57233b7dafe9bef0008fd5d3575cd7675a6f43d4df02412836f63ede8e3a3a"} Aug 04 10:58:04.458326 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:04.458288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" event={"ID":"1038c84e-5e62-425c-addd-b1110c2a6e7e","Type":"ContainerStarted","Data":"d049fd9b90746d7e448082833a019d49d711d09889c0f4a59d458ac29f3a8821"} Aug 04 10:58:04.458777 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:04.458479 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:04.475874 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:04.475813 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" podStartSLOduration=2.05925447 podStartE2EDuration="2.475800696s" totalImagesPullingTime="416.546226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:58:04.474249431 +0000 UTC m=+1845.425656979" watchObservedRunningTime="2026-08-04 10:58:04.475800696 +0000 UTC m=+1845.427208279" Aug 04 10:58:10.469473 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:10.469441 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:32.650015 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.649984 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx"] Aug 04 10:58:32.653566 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.653550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" Aug 04 10:58:32.656008 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.655987 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 10:58:32.662162 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.662137 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx"] Aug 04 10:58:32.808424 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.808393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767527be-1c1f-42a3-ada0-8189f3128cae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx\" (UID: \"767527be-1c1f-42a3-ada0-8189f3128cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" Aug 04 10:58:32.908814 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.908745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767527be-1c1f-42a3-ada0-8189f3128cae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx\" (UID: \"767527be-1c1f-42a3-ada0-8189f3128cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" Aug 04 10:58:32.909350 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.909332 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767527be-1c1f-42a3-ada0-8189f3128cae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx\" (UID: \"767527be-1c1f-42a3-ada0-8189f3128cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" Aug 04 10:58:32.964906 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:32.964884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" Aug 04 10:58:33.098078 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:33.098022 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx"] Aug 04 10:58:33.100350 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:58:33.100321 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767527be_1c1f_42a3_ada0_8189f3128cae.slice/crio-d150fe1cf8bbf48cdadff5aad6e91cbe935e9a0734b6a8b7086875ea5e7c3854 WatchSource:0}: Error finding container d150fe1cf8bbf48cdadff5aad6e91cbe935e9a0734b6a8b7086875ea5e7c3854: Status 404 returned error can't find the container with id d150fe1cf8bbf48cdadff5aad6e91cbe935e9a0734b6a8b7086875ea5e7c3854 Aug 04 10:58:33.610466 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:33.610386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" event={"ID":"767527be-1c1f-42a3-ada0-8189f3128cae","Type":"ContainerStarted","Data":"0d21c736af7168c98501f06808182df86dfe5ffd2fa904924422920226bf4663"} Aug 04 10:58:33.610466 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:33.610442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" event={"ID":"767527be-1c1f-42a3-ada0-8189f3128cae","Type":"ContainerStarted","Data":"d150fe1cf8bbf48cdadff5aad6e91cbe935e9a0734b6a8b7086875ea5e7c3854"} Aug 04 10:58:33.625872 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:33.625809 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" podStartSLOduration=1.625791392 podStartE2EDuration="1.625791392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:58:33.625051359 +0000 UTC m=+1874.576458910" watchObservedRunningTime="2026-08-04 10:58:33.625791392 +0000 UTC m=+1874.577198941" Aug 04 10:58:34.616675 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:34.616641 2577 generic.go:356] "Generic (PLEG): container finished" podID="767527be-1c1f-42a3-ada0-8189f3128cae" containerID="0d21c736af7168c98501f06808182df86dfe5ffd2fa904924422920226bf4663" exitCode=0 Aug 04 10:58:34.617078 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:34.616722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" event={"ID":"767527be-1c1f-42a3-ada0-8189f3128cae","Type":"ContainerDied","Data":"0d21c736af7168c98501f06808182df86dfe5ffd2fa904924422920226bf4663"} Aug 04 10:58:35.747022 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:35.746999 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" Aug 04 10:58:35.833478 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:35.833451 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767527be-1c1f-42a3-ada0-8189f3128cae-maas-api-service-ca\") pod \"767527be-1c1f-42a3-ada0-8189f3128cae\" (UID: \"767527be-1c1f-42a3-ada0-8189f3128cae\") " Aug 04 10:58:35.833759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:35.833737 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767527be-1c1f-42a3-ada0-8189f3128cae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "767527be-1c1f-42a3-ada0-8189f3128cae" (UID: "767527be-1c1f-42a3-ada0-8189f3128cae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:58:35.935154 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:35.935099 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767527be-1c1f-42a3-ada0-8189f3128cae-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:58:36.628414 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:36.628384 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" Aug 04 10:58:36.628581 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:36.628415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx" event={"ID":"767527be-1c1f-42a3-ada0-8189f3128cae","Type":"ContainerDied","Data":"d150fe1cf8bbf48cdadff5aad6e91cbe935e9a0734b6a8b7086875ea5e7c3854"} Aug 04 10:58:36.628581 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:36.628462 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d150fe1cf8bbf48cdadff5aad6e91cbe935e9a0734b6a8b7086875ea5e7c3854" Aug 04 10:58:43.423056 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.422982 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p"] Aug 04 10:58:43.423505 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.423200 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" podUID="1038c84e-5e62-425c-addd-b1110c2a6e7e" containerName="maas-api" containerID="cri-o://d049fd9b90746d7e448082833a019d49d711d09889c0f4a59d458ac29f3a8821" gracePeriod=30 Aug 04 10:58:43.665876 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.665822 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx"] Aug 04 10:58:43.668167 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.668141 2577 generic.go:356] "Generic (PLEG): container finished" podID="1038c84e-5e62-425c-addd-b1110c2a6e7e" containerID="d049fd9b90746d7e448082833a019d49d711d09889c0f4a59d458ac29f3a8821" exitCode=0 Aug 04 10:58:43.668285 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.668191 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" event={"ID":"1038c84e-5e62-425c-addd-b1110c2a6e7e","Type":"ContainerDied","Data":"d049fd9b90746d7e448082833a019d49d711d09889c0f4a59d458ac29f3a8821"} Aug 04 10:58:43.668285 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.668240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" event={"ID":"1038c84e-5e62-425c-addd-b1110c2a6e7e","Type":"ContainerDied","Data":"1e57233b7dafe9bef0008fd5d3575cd7675a6f43d4df02412836f63ede8e3a3a"} Aug 04 10:58:43.668285 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.668261 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e57233b7dafe9bef0008fd5d3575cd7675a6f43d4df02412836f63ede8e3a3a" Aug 04 10:58:43.668520 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.668499 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb58dd84-k5gsx"] Aug 04 10:58:43.670466 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.670450 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:43.809551 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.809517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgtvv\" (UniqueName: \"kubernetes.io/projected/1038c84e-5e62-425c-addd-b1110c2a6e7e-kube-api-access-kgtvv\") pod \"1038c84e-5e62-425c-addd-b1110c2a6e7e\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " Aug 04 10:58:43.809712 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.809564 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1038c84e-5e62-425c-addd-b1110c2a6e7e-openshift-service-ca\") pod \"1038c84e-5e62-425c-addd-b1110c2a6e7e\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " Aug 04 10:58:43.809712 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.809595 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1038c84e-5e62-425c-addd-b1110c2a6e7e-maas-api-tls\") pod \"1038c84e-5e62-425c-addd-b1110c2a6e7e\" (UID: \"1038c84e-5e62-425c-addd-b1110c2a6e7e\") " Aug 04 10:58:43.810000 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.809973 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1038c84e-5e62-425c-addd-b1110c2a6e7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1038c84e-5e62-425c-addd-b1110c2a6e7e" (UID: "1038c84e-5e62-425c-addd-b1110c2a6e7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:58:43.811628 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.811607 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1038c84e-5e62-425c-addd-b1110c2a6e7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1038c84e-5e62-425c-addd-b1110c2a6e7e" (UID: "1038c84e-5e62-425c-addd-b1110c2a6e7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:58:43.811738 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.811632 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1038c84e-5e62-425c-addd-b1110c2a6e7e-kube-api-access-kgtvv" (OuterVolumeSpecName: "kube-api-access-kgtvv") pod "1038c84e-5e62-425c-addd-b1110c2a6e7e" (UID: "1038c84e-5e62-425c-addd-b1110c2a6e7e"). InnerVolumeSpecName "kube-api-access-kgtvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:58:43.910437 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.910415 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1038c84e-5e62-425c-addd-b1110c2a6e7e-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:58:43.910437 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.910437 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgtvv\" (UniqueName: \"kubernetes.io/projected/1038c84e-5e62-425c-addd-b1110c2a6e7e-kube-api-access-kgtvv\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:58:43.910583 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:43.910452 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1038c84e-5e62-425c-addd-b1110c2a6e7e-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:58:44.674052 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:44.674022 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p" Aug 04 10:58:44.696673 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:44.696645 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p"] Aug 04 10:58:44.698812 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:44.698792 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fb58dd84-79d45c8-9sg2p"] Aug 04 10:58:45.532395 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:45.532363 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1038c84e-5e62-425c-addd-b1110c2a6e7e" path="/var/lib/kubelet/pods/1038c84e-5e62-425c-addd-b1110c2a6e7e/volumes" Aug 04 10:58:45.532674 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:45.532661 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="767527be-1c1f-42a3-ada0-8189f3128cae" path="/var/lib/kubelet/pods/767527be-1c1f-42a3-ada0-8189f3128cae/volumes" Aug 04 10:58:49.125599 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.125565 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf"] Aug 04 10:58:49.128067 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.125975 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767527be-1c1f-42a3-ada0-8189f3128cae" containerName="revoke-keys" Aug 04 10:58:49.128067 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.125989 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="767527be-1c1f-42a3-ada0-8189f3128cae" containerName="revoke-keys" Aug 04 10:58:49.128067 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.126002 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1038c84e-5e62-425c-addd-b1110c2a6e7e" containerName="maas-api" Aug 04 10:58:49.128067 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.126007 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1038c84e-5e62-425c-addd-b1110c2a6e7e" containerName="maas-api" Aug 04 10:58:49.128067 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.126074 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1038c84e-5e62-425c-addd-b1110c2a6e7e" containerName="maas-api" Aug 04 10:58:49.128067 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.126085 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="767527be-1c1f-42a3-ada0-8189f3128cae" containerName="revoke-keys" Aug 04 10:58:49.128970 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.128952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.130994 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.130973 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-524f2cac-serving-cert\"" Aug 04 10:58:49.139255 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.139232 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf"] Aug 04 10:58:49.155307 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.155284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl429\" (UniqueName: \"kubernetes.io/projected/7bd14e88-7a87-4e23-9f47-cd04ea741abe-kube-api-access-vl429\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.155415 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.155318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd14e88-7a87-4e23-9f47-cd04ea741abe-openshift-service-ca\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.155415 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.155393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.255934 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.255894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl429\" (UniqueName: \"kubernetes.io/projected/7bd14e88-7a87-4e23-9f47-cd04ea741abe-kube-api-access-vl429\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.255934 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.255934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd14e88-7a87-4e23-9f47-cd04ea741abe-openshift-service-ca\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.256153 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.256026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.256198 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:58:49.256162 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-serving-cert: secret "maas-api-e2e-derive-524f2cac-serving-cert" not found Aug 04 10:58:49.256238 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:58:49.256228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls podName:7bd14e88-7a87-4e23-9f47-cd04ea741abe nodeName:}" failed. No retries permitted until 2026-08-04 10:58:49.756209738 +0000 UTC m=+1890.707617279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls") pod "maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" (UID: "7bd14e88-7a87-4e23-9f47-cd04ea741abe") : secret "maas-api-e2e-derive-524f2cac-serving-cert" not found Aug 04 10:58:49.256627 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.256607 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd14e88-7a87-4e23-9f47-cd04ea741abe-openshift-service-ca\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.265031 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.265005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl429\" (UniqueName: \"kubernetes.io/projected/7bd14e88-7a87-4e23-9f47-cd04ea741abe-kube-api-access-vl429\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.759651 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.759601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:49.761981 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:49.761951 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls\") pod \"maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:50.040065 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:50.039986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:50.181129 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:50.181025 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf"] Aug 04 10:58:50.183472 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:58:50.183443 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd14e88_7a87_4e23_9f47_cd04ea741abe.slice/crio-75f37989781eebd58b379c40b82b480d26ceec6af39b5354796c8c15ca27f771 WatchSource:0}: Error finding container 75f37989781eebd58b379c40b82b480d26ceec6af39b5354796c8c15ca27f771: Status 404 returned error can't find the container with id 75f37989781eebd58b379c40b82b480d26ceec6af39b5354796c8c15ca27f771 Aug 04 10:58:50.710097 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:50.710057 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" event={"ID":"7bd14e88-7a87-4e23-9f47-cd04ea741abe","Type":"ContainerStarted","Data":"13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d"} Aug 04 10:58:50.710283 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:50.710126 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" event={"ID":"7bd14e88-7a87-4e23-9f47-cd04ea741abe","Type":"ContainerStarted","Data":"75f37989781eebd58b379c40b82b480d26ceec6af39b5354796c8c15ca27f771"} Aug 04 10:58:50.710283 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:50.710155 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:58:50.729785 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:50.729737 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" podStartSLOduration=1.4261454900000001 podStartE2EDuration="1.729719374s" totalImagesPullingTime="303.573884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:58:50.727501003 +0000 UTC m=+1891.678908552" watchObservedRunningTime="2026-08-04 10:58:50.729719374 +0000 UTC m=+1891.681126943" Aug 04 10:58:56.721474 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:58:56.721444 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:59:12.789225 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:12.789192 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z"] Aug 04 10:59:12.792751 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:12.792734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" Aug 04 10:59:12.794952 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:12.794926 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 10:59:12.801294 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:12.800979 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z"] Aug 04 10:59:12.874173 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:12.874142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9806e41-de9d-41c1-a17d-7f21d3109819-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-524f2cac-52z7z\" (UID: \"e9806e41-de9d-41c1-a17d-7f21d3109819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" Aug 04 10:59:12.975003 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:12.974965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9806e41-de9d-41c1-a17d-7f21d3109819-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-524f2cac-52z7z\" (UID: \"e9806e41-de9d-41c1-a17d-7f21d3109819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" Aug 04 10:59:12.975594 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:12.975576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9806e41-de9d-41c1-a17d-7f21d3109819-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-524f2cac-52z7z\" (UID: \"e9806e41-de9d-41c1-a17d-7f21d3109819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" Aug 04 10:59:13.105106 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:13.105026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" Aug 04 10:59:13.247763 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:13.247735 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z"] Aug 04 10:59:13.249935 ip-10-0-139-198 kubenswrapper[2577]: W0804 10:59:13.249899 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9806e41_de9d_41c1_a17d_7f21d3109819.slice/crio-d8d6bf46631a7e74bbc8f4681ff3a127edbcae32c652821d86191441c4c3725d WatchSource:0}: Error finding container d8d6bf46631a7e74bbc8f4681ff3a127edbcae32c652821d86191441c4c3725d: Status 404 returned error can't find the container with id d8d6bf46631a7e74bbc8f4681ff3a127edbcae32c652821d86191441c4c3725d Aug 04 10:59:13.849074 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:13.849034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" event={"ID":"e9806e41-de9d-41c1-a17d-7f21d3109819","Type":"ContainerStarted","Data":"0fc765e5bacb87c32f581dd79fd8408d0fac0e236d4265849c3333ba44065aa7"} Aug 04 10:59:13.849494 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:13.849087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" event={"ID":"e9806e41-de9d-41c1-a17d-7f21d3109819","Type":"ContainerStarted","Data":"d8d6bf46631a7e74bbc8f4681ff3a127edbcae32c652821d86191441c4c3725d"} Aug 04 10:59:13.865923 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:13.865872 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" podStartSLOduration=1.865857349 podStartE2EDuration="1.865857349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 10:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 10:59:13.864582368 +0000 UTC m=+1914.815989932" watchObservedRunningTime="2026-08-04 10:59:13.865857349 +0000 UTC m=+1914.817264899" Aug 04 10:59:14.855644 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:14.855610 2577 generic.go:356] "Generic (PLEG): container finished" podID="e9806e41-de9d-41c1-a17d-7f21d3109819" containerID="0fc765e5bacb87c32f581dd79fd8408d0fac0e236d4265849c3333ba44065aa7" exitCode=0 Aug 04 10:59:14.856070 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:14.855685 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" event={"ID":"e9806e41-de9d-41c1-a17d-7f21d3109819","Type":"ContainerDied","Data":"0fc765e5bacb87c32f581dd79fd8408d0fac0e236d4265849c3333ba44065aa7"} Aug 04 10:59:15.990161 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:15.990137 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" Aug 04 10:59:16.101498 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:16.101459 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9806e41-de9d-41c1-a17d-7f21d3109819-maas-api-service-ca\") pod \"e9806e41-de9d-41c1-a17d-7f21d3109819\" (UID: \"e9806e41-de9d-41c1-a17d-7f21d3109819\") " Aug 04 10:59:16.101794 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:16.101773 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9806e41-de9d-41c1-a17d-7f21d3109819-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9806e41-de9d-41c1-a17d-7f21d3109819" (UID: "e9806e41-de9d-41c1-a17d-7f21d3109819"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:59:16.202643 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:16.202562 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9806e41-de9d-41c1-a17d-7f21d3109819-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:59:16.868416 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:16.868388 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" Aug 04 10:59:16.868592 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:16.868385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z" event={"ID":"e9806e41-de9d-41c1-a17d-7f21d3109819","Type":"ContainerDied","Data":"d8d6bf46631a7e74bbc8f4681ff3a127edbcae32c652821d86191441c4c3725d"} Aug 04 10:59:16.868592 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:16.868517 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8d6bf46631a7e74bbc8f4681ff3a127edbcae32c652821d86191441c4c3725d" Aug 04 10:59:19.566423 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.566386 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf"] Aug 04 10:59:19.566864 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.566582 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" podUID="7bd14e88-7a87-4e23-9f47-cd04ea741abe" containerName="maas-api" containerID="cri-o://13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d" gracePeriod=30 Aug 04 10:59:19.769161 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.769126 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z"] Aug 04 10:59:19.771340 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.771309 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-524f2cac-52z7z"] Aug 04 10:59:19.817492 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.817439 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:59:19.889566 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.889528 2577 generic.go:356] "Generic (PLEG): container finished" podID="7bd14e88-7a87-4e23-9f47-cd04ea741abe" containerID="13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d" exitCode=0 Aug 04 10:59:19.889759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.889589 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" Aug 04 10:59:19.889759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.889605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" event={"ID":"7bd14e88-7a87-4e23-9f47-cd04ea741abe","Type":"ContainerDied","Data":"13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d"} Aug 04 10:59:19.889759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.889698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf" event={"ID":"7bd14e88-7a87-4e23-9f47-cd04ea741abe","Type":"ContainerDied","Data":"75f37989781eebd58b379c40b82b480d26ceec6af39b5354796c8c15ca27f771"} Aug 04 10:59:19.889759 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.889732 2577 scope.go:117] "RemoveContainer" containerID="13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d" Aug 04 10:59:19.902642 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.902618 2577 scope.go:117] "RemoveContainer" containerID="13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d" Aug 04 10:59:19.903145 ip-10-0-139-198 kubenswrapper[2577]: E0804 10:59:19.903123 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d\": container with ID starting with 13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d not found: ID does not exist" containerID="13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d" Aug 04 10:59:19.903230 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.903159 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d"} err="failed to get container status \"13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d\": rpc error: code = NotFound desc = could not find container \"13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d\": container with ID starting with 13eca3c865030686187b146516acf209e53087a270b863d67eace861b0f86b3d not found: ID does not exist" Aug 04 10:59:19.938130 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.938101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls\") pod \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " Aug 04 10:59:19.938280 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.938226 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl429\" (UniqueName: \"kubernetes.io/projected/7bd14e88-7a87-4e23-9f47-cd04ea741abe-kube-api-access-vl429\") pod \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " Aug 04 10:59:19.938280 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.938253 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd14e88-7a87-4e23-9f47-cd04ea741abe-openshift-service-ca\") pod \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\" (UID: \"7bd14e88-7a87-4e23-9f47-cd04ea741abe\") " Aug 04 10:59:19.938651 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.938625 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd14e88-7a87-4e23-9f47-cd04ea741abe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bd14e88-7a87-4e23-9f47-cd04ea741abe" (UID: "7bd14e88-7a87-4e23-9f47-cd04ea741abe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 10:59:19.940187 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.940159 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bd14e88-7a87-4e23-9f47-cd04ea741abe" (UID: "7bd14e88-7a87-4e23-9f47-cd04ea741abe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 10:59:19.940370 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:19.940352 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd14e88-7a87-4e23-9f47-cd04ea741abe-kube-api-access-vl429" (OuterVolumeSpecName: "kube-api-access-vl429") pod "7bd14e88-7a87-4e23-9f47-cd04ea741abe" (UID: "7bd14e88-7a87-4e23-9f47-cd04ea741abe"). InnerVolumeSpecName "kube-api-access-vl429". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 10:59:20.039917 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:20.039877 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bd14e88-7a87-4e23-9f47-cd04ea741abe-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:59:20.039917 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:20.039910 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl429\" (UniqueName: \"kubernetes.io/projected/7bd14e88-7a87-4e23-9f47-cd04ea741abe-kube-api-access-vl429\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:59:20.039917 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:20.039922 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd14e88-7a87-4e23-9f47-cd04ea741abe-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 10:59:20.210599 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:20.210565 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf"] Aug 04 10:59:20.216715 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:20.216674 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-524f2cac-6cbdbcf4cd-c4cxf"] Aug 04 10:59:21.533258 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:21.533221 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd14e88-7a87-4e23-9f47-cd04ea741abe" path="/var/lib/kubelet/pods/7bd14e88-7a87-4e23-9f47-cd04ea741abe/volumes" Aug 04 10:59:21.533643 ip-10-0-139-198 kubenswrapper[2577]: I0804 10:59:21.533560 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9806e41-de9d-41c1-a17d-7f21d3109819" path="/var/lib/kubelet/pods/e9806e41-de9d-41c1-a17d-7f21d3109819/volumes" Aug 04 11:01:26.717793 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.717754 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm"] Aug 04 11:01:26.718300 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.718147 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd14e88-7a87-4e23-9f47-cd04ea741abe" containerName="maas-api" Aug 04 11:01:26.718300 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.718159 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd14e88-7a87-4e23-9f47-cd04ea741abe" containerName="maas-api" Aug 04 11:01:26.718300 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.718170 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9806e41-de9d-41c1-a17d-7f21d3109819" containerName="revoke-keys" Aug 04 11:01:26.718300 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.718176 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9806e41-de9d-41c1-a17d-7f21d3109819" containerName="revoke-keys" Aug 04 11:01:26.718300 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.718252 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9806e41-de9d-41c1-a17d-7f21d3109819" containerName="revoke-keys" Aug 04 11:01:26.718300 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.718264 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd14e88-7a87-4e23-9f47-cd04ea741abe" containerName="maas-api" Aug 04 11:01:26.721363 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.721342 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.723147 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.723129 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fe15eb98-serving-cert\"" Aug 04 11:01:26.730921 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.730901 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm"] Aug 04 11:01:26.773865 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.773813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6m5v\" (UniqueName: \"kubernetes.io/projected/a2297607-8be8-42ed-8858-a6770909a9d5-kube-api-access-w6m5v\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.773991 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.773948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.774059 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.774001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2297607-8be8-42ed-8858-a6770909a9d5-openshift-service-ca\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.875114 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.875075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6m5v\" (UniqueName: \"kubernetes.io/projected/a2297607-8be8-42ed-8858-a6770909a9d5-kube-api-access-w6m5v\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.875298 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.875150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.875298 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.875183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2297607-8be8-42ed-8858-a6770909a9d5-openshift-service-ca\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.875298 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:01:26.875290 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-serving-cert: secret "maas-api-e2e-mt-fe15eb98-serving-cert" not found Aug 04 11:01:26.875478 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:01:26.875354 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls podName:a2297607-8be8-42ed-8858-a6770909a9d5 nodeName:}" failed. No retries permitted until 2026-08-04 11:01:27.375334758 +0000 UTC m=+2048.326742286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls") pod "maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" (UID: "a2297607-8be8-42ed-8858-a6770909a9d5") : secret "maas-api-e2e-mt-fe15eb98-serving-cert" not found Aug 04 11:01:26.875781 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.875764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2297607-8be8-42ed-8858-a6770909a9d5-openshift-service-ca\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:26.882870 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:26.882847 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6m5v\" (UniqueName: \"kubernetes.io/projected/a2297607-8be8-42ed-8858-a6770909a9d5-kube-api-access-w6m5v\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:27.380621 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:27.380585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:27.383191 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:27.383163 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls\") pod \"maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:27.632601 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:27.632523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:27.772636 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:27.772611 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm"] Aug 04 11:01:27.774886 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:01:27.774861 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2297607_8be8_42ed_8858_a6770909a9d5.slice/crio-e298bd64ed30c15cde1a1cdad92d10c9b3ccb1882a0e0496ac49a80eea9cfb55 WatchSource:0}: Error finding container e298bd64ed30c15cde1a1cdad92d10c9b3ccb1882a0e0496ac49a80eea9cfb55: Status 404 returned error can't find the container with id e298bd64ed30c15cde1a1cdad92d10c9b3ccb1882a0e0496ac49a80eea9cfb55 Aug 04 11:01:28.625428 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:28.625342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" event={"ID":"a2297607-8be8-42ed-8858-a6770909a9d5","Type":"ContainerStarted","Data":"8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec"} Aug 04 11:01:28.625428 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:28.625420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" event={"ID":"a2297607-8be8-42ed-8858-a6770909a9d5","Type":"ContainerStarted","Data":"e298bd64ed30c15cde1a1cdad92d10c9b3ccb1882a0e0496ac49a80eea9cfb55"} Aug 04 11:01:28.625607 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:28.625442 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:28.645480 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:28.645430 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" podStartSLOduration=2.151294417 podStartE2EDuration="2.645415983s" totalImagesPullingTime="494.121566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:01:28.643009716 +0000 UTC m=+2049.594417266" watchObservedRunningTime="2026-08-04 11:01:28.645415983 +0000 UTC m=+2049.596823578" Aug 04 11:01:34.636309 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:34.636278 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:48.349473 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.349387 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c"] Aug 04 11:01:48.353283 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.353259 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" Aug 04 11:01:48.355050 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.355031 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 11:01:48.360524 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.360470 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c"] Aug 04 11:01:48.473169 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.473137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae884e-e992-4025-9821-2b1c5c768c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c\" (UID: \"7fae884e-e992-4025-9821-2b1c5c768c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" Aug 04 11:01:48.573731 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.573696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae884e-e992-4025-9821-2b1c5c768c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c\" (UID: \"7fae884e-e992-4025-9821-2b1c5c768c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" Aug 04 11:01:48.574341 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.574320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae884e-e992-4025-9821-2b1c5c768c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c\" (UID: \"7fae884e-e992-4025-9821-2b1c5c768c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" Aug 04 11:01:48.666468 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.666389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" Aug 04 11:01:48.801861 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:48.801812 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c"] Aug 04 11:01:48.803899 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:01:48.803873 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae884e_e992_4025_9821_2b1c5c768c62.slice/crio-8d6b679af80c829b825b8007bfa7775c3c0e62e7a20a62dcead1c81e77adc0a3 WatchSource:0}: Error finding container 8d6b679af80c829b825b8007bfa7775c3c0e62e7a20a62dcead1c81e77adc0a3: Status 404 returned error can't find the container with id 8d6b679af80c829b825b8007bfa7775c3c0e62e7a20a62dcead1c81e77adc0a3 Aug 04 11:01:49.742427 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:49.742339 2577 generic.go:356] "Generic (PLEG): container finished" podID="7fae884e-e992-4025-9821-2b1c5c768c62" containerID="256bf958da45c0cd2715090500c6cbe478bb4932c3ee10ece519b0224c5c6576" exitCode=0 Aug 04 11:01:49.742777 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:49.742439 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" event={"ID":"7fae884e-e992-4025-9821-2b1c5c768c62","Type":"ContainerDied","Data":"256bf958da45c0cd2715090500c6cbe478bb4932c3ee10ece519b0224c5c6576"} Aug 04 11:01:49.742777 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:49.742488 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" event={"ID":"7fae884e-e992-4025-9821-2b1c5c768c62","Type":"ContainerStarted","Data":"8d6b679af80c829b825b8007bfa7775c3c0e62e7a20a62dcead1c81e77adc0a3"} Aug 04 11:01:50.881289 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:50.881264 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" Aug 04 11:01:50.998730 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:50.998644 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae884e-e992-4025-9821-2b1c5c768c62-maas-api-service-ca\") pod \"7fae884e-e992-4025-9821-2b1c5c768c62\" (UID: \"7fae884e-e992-4025-9821-2b1c5c768c62\") " Aug 04 11:01:50.998969 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:50.998948 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fae884e-e992-4025-9821-2b1c5c768c62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fae884e-e992-4025-9821-2b1c5c768c62" (UID: "7fae884e-e992-4025-9821-2b1c5c768c62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:01:51.099616 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:51.099590 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fae884e-e992-4025-9821-2b1c5c768c62-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:01:51.754597 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:51.754565 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" Aug 04 11:01:51.754597 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:51.754585 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c" event={"ID":"7fae884e-e992-4025-9821-2b1c5c768c62","Type":"ContainerDied","Data":"8d6b679af80c829b825b8007bfa7775c3c0e62e7a20a62dcead1c81e77adc0a3"} Aug 04 11:01:51.754867 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:51.754631 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d6b679af80c829b825b8007bfa7775c3c0e62e7a20a62dcead1c81e77adc0a3" Aug 04 11:01:53.753654 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:53.753615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm"] Aug 04 11:01:53.754127 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:53.753887 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" podUID="a2297607-8be8-42ed-8858-a6770909a9d5" containerName="maas-api" containerID="cri-o://8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec" gracePeriod=30 Aug 04 11:01:53.996476 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:53.996450 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:54.108647 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.108536 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c"] Aug 04 11:01:54.110944 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.110915 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe15eb98-jms7c"] Aug 04 11:01:54.130309 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.130275 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls\") pod \"a2297607-8be8-42ed-8858-a6770909a9d5\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " Aug 04 11:01:54.130462 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.130331 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2297607-8be8-42ed-8858-a6770909a9d5-openshift-service-ca\") pod \"a2297607-8be8-42ed-8858-a6770909a9d5\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " Aug 04 11:01:54.130462 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.130359 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6m5v\" (UniqueName: \"kubernetes.io/projected/a2297607-8be8-42ed-8858-a6770909a9d5-kube-api-access-w6m5v\") pod \"a2297607-8be8-42ed-8858-a6770909a9d5\" (UID: \"a2297607-8be8-42ed-8858-a6770909a9d5\") " Aug 04 11:01:54.130791 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.130755 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2297607-8be8-42ed-8858-a6770909a9d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2297607-8be8-42ed-8858-a6770909a9d5" (UID: "a2297607-8be8-42ed-8858-a6770909a9d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:01:54.132448 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.132423 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2297607-8be8-42ed-8858-a6770909a9d5" (UID: "a2297607-8be8-42ed-8858-a6770909a9d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:01:54.132448 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.132438 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2297607-8be8-42ed-8858-a6770909a9d5-kube-api-access-w6m5v" (OuterVolumeSpecName: "kube-api-access-w6m5v") pod "a2297607-8be8-42ed-8858-a6770909a9d5" (UID: "a2297607-8be8-42ed-8858-a6770909a9d5"). InnerVolumeSpecName "kube-api-access-w6m5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:01:54.231417 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.231375 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2297607-8be8-42ed-8858-a6770909a9d5-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:01:54.231417 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.231411 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2297607-8be8-42ed-8858-a6770909a9d5-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:01:54.231417 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.231422 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6m5v\" (UniqueName: \"kubernetes.io/projected/a2297607-8be8-42ed-8858-a6770909a9d5-kube-api-access-w6m5v\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:01:54.775741 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.775703 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2297607-8be8-42ed-8858-a6770909a9d5" containerID="8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec" exitCode=0 Aug 04 11:01:54.776184 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.775759 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" Aug 04 11:01:54.776184 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.775795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" event={"ID":"a2297607-8be8-42ed-8858-a6770909a9d5","Type":"ContainerDied","Data":"8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec"} Aug 04 11:01:54.776184 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.775863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm" event={"ID":"a2297607-8be8-42ed-8858-a6770909a9d5","Type":"ContainerDied","Data":"e298bd64ed30c15cde1a1cdad92d10c9b3ccb1882a0e0496ac49a80eea9cfb55"} Aug 04 11:01:54.776184 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.775888 2577 scope.go:117] "RemoveContainer" containerID="8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec" Aug 04 11:01:54.788248 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.788226 2577 scope.go:117] "RemoveContainer" containerID="8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec" Aug 04 11:01:54.788498 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:01:54.788478 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec\": container with ID starting with 8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec not found: ID does not exist" containerID="8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec" Aug 04 11:01:54.788538 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.788507 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec"} err="failed to get container status \"8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec\": rpc error: code = NotFound desc = could not find container \"8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec\": container with ID starting with 8883ff953ae7e3e5cc7312091bc03338143767ca3e8d4f3a64262fde1adee2ec not found: ID does not exist" Aug 04 11:01:54.799150 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.799124 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm"] Aug 04 11:01:54.801412 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:54.801388 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe15eb98-68ccf6fc67-plgrm"] Aug 04 11:01:55.533087 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:55.533048 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fae884e-e992-4025-9821-2b1c5c768c62" path="/var/lib/kubelet/pods/7fae884e-e992-4025-9821-2b1c5c768c62/volumes" Aug 04 11:01:55.533534 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:01:55.533514 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2297607-8be8-42ed-8858-a6770909a9d5" path="/var/lib/kubelet/pods/a2297607-8be8-42ed-8858-a6770909a9d5/volumes" Aug 04 11:02:19.728451 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:02:19.728420 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:02:19.736638 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:02:19.736615 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:03:19.786753 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:03:19.786719 2577 scope.go:117] "RemoveContainer" containerID="e744746a7a7a3068609b945e5b44128659f1df622134d22733e2015c017b7785" Aug 04 11:04:19.809360 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:19.809327 2577 scope.go:117] "RemoveContainer" containerID="d049fd9b90746d7e448082833a019d49d711d09889c0f4a59d458ac29f3a8821" Aug 04 11:04:19.818270 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:19.818245 2577 scope.go:117] "RemoveContainer" containerID="8218f21532550d5384ddd623efef434e39521635da1cfa6e1306b28c5221d862" Aug 04 11:04:22.158229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.158194 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf"] Aug 04 11:04:22.158678 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.158574 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2297607-8be8-42ed-8858-a6770909a9d5" containerName="maas-api" Aug 04 11:04:22.158678 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.158587 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2297607-8be8-42ed-8858-a6770909a9d5" containerName="maas-api" Aug 04 11:04:22.158678 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.158596 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fae884e-e992-4025-9821-2b1c5c768c62" containerName="revoke-keys" Aug 04 11:04:22.158678 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.158601 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fae884e-e992-4025-9821-2b1c5c768c62" containerName="revoke-keys" Aug 04 11:04:22.158925 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.158691 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2297607-8be8-42ed-8858-a6770909a9d5" containerName="maas-api" Aug 04 11:04:22.158925 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.158699 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fae884e-e992-4025-9821-2b1c5c768c62" containerName="revoke-keys" Aug 04 11:04:22.163225 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.163208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.165335 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.165318 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a18af8fb-serving-cert\"" Aug 04 11:04:22.171788 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.171706 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf"] Aug 04 11:04:22.258723 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.258690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.258894 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.258736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f77950-dee2-4c1f-a6a9-5d15949093f0-openshift-service-ca\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.258894 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.258752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbw42\" (UniqueName: \"kubernetes.io/projected/c8f77950-dee2-4c1f-a6a9-5d15949093f0-kube-api-access-dbw42\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.359757 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.359716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.359757 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.359762 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f77950-dee2-4c1f-a6a9-5d15949093f0-openshift-service-ca\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.360032 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.359780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbw42\" (UniqueName: \"kubernetes.io/projected/c8f77950-dee2-4c1f-a6a9-5d15949093f0-kube-api-access-dbw42\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.360032 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:04:22.359880 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-serving-cert: secret "maas-api-e2e-mt-a18af8fb-serving-cert" not found Aug 04 11:04:22.360032 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:04:22.359947 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls podName:c8f77950-dee2-4c1f-a6a9-5d15949093f0 nodeName:}" failed. No retries permitted until 2026-08-04 11:04:22.85993081 +0000 UTC m=+2223.811338337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls") pod "maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" (UID: "c8f77950-dee2-4c1f-a6a9-5d15949093f0") : secret "maas-api-e2e-mt-a18af8fb-serving-cert" not found Aug 04 11:04:22.360456 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.360437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f77950-dee2-4c1f-a6a9-5d15949093f0-openshift-service-ca\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.369229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.369201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbw42\" (UniqueName: \"kubernetes.io/projected/c8f77950-dee2-4c1f-a6a9-5d15949093f0-kube-api-access-dbw42\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.864736 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.864703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:22.867303 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:22.867280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls\") pod \"maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:23.074424 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:23.074393 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:23.211620 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:23.211591 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf"] Aug 04 11:04:23.213816 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:04:23.213791 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f77950_dee2_4c1f_a6a9_5d15949093f0.slice/crio-728dbfde5955807c4cd0084276ca397273638f12b1ea7c6c6c6d0e8e0c589cdd WatchSource:0}: Error finding container 728dbfde5955807c4cd0084276ca397273638f12b1ea7c6c6c6d0e8e0c589cdd: Status 404 returned error can't find the container with id 728dbfde5955807c4cd0084276ca397273638f12b1ea7c6c6c6d0e8e0c589cdd Aug 04 11:04:23.215221 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:23.215204 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:04:23.624796 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:23.624760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" event={"ID":"c8f77950-dee2-4c1f-a6a9-5d15949093f0","Type":"ContainerStarted","Data":"728dbfde5955807c4cd0084276ca397273638f12b1ea7c6c6c6d0e8e0c589cdd"} Aug 04 11:04:26.644484 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:26.644445 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" event={"ID":"c8f77950-dee2-4c1f-a6a9-5d15949093f0","Type":"ContainerStarted","Data":"474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42"} Aug 04 11:04:26.644948 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:26.644603 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:04:26.662570 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:26.662528 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" podStartSLOduration=2.263658214 podStartE2EDuration="4.662516187s" totalImagesPullingTime="2.398857973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:04:26.661282807 +0000 UTC m=+2227.612690355" watchObservedRunningTime="2026-08-04 11:04:26.662516187 +0000 UTC m=+2227.613923735" Aug 04 11:04:32.656021 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:04:32.655944 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:05:19.841750 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:19.841720 2577 scope.go:117] "RemoveContainer" containerID="0d21c736af7168c98501f06808182df86dfe5ffd2fa904924422920226bf4663" Aug 04 11:05:19.850092 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:19.850072 2577 scope.go:117] "RemoveContainer" containerID="0fc765e5bacb87c32f581dd79fd8408d0fac0e236d4265849c3333ba44065aa7" Aug 04 11:05:30.899660 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:30.899624 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr"] Aug 04 11:05:30.903123 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:30.903106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" Aug 04 11:05:30.905245 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:30.905226 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 11:05:30.913688 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:30.913666 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr"] Aug 04 11:05:30.960879 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:30.960851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74965014-c6d0-4178-b6ef-55e068bc8506-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr\" (UID: \"74965014-c6d0-4178-b6ef-55e068bc8506\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" Aug 04 11:05:31.061969 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:31.061942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74965014-c6d0-4178-b6ef-55e068bc8506-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr\" (UID: \"74965014-c6d0-4178-b6ef-55e068bc8506\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" Aug 04 11:05:31.062532 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:31.062513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74965014-c6d0-4178-b6ef-55e068bc8506-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr\" (UID: \"74965014-c6d0-4178-b6ef-55e068bc8506\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" Aug 04 11:05:31.213651 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:31.213571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" Aug 04 11:05:31.342852 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:31.342813 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr"] Aug 04 11:05:31.345202 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:05:31.345173 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74965014_c6d0_4178_b6ef_55e068bc8506.slice/crio-db21ad1175c5bf7afffe6a8a269ab6c438cdbba805d8ba22a642272309ec1e0c WatchSource:0}: Error finding container db21ad1175c5bf7afffe6a8a269ab6c438cdbba805d8ba22a642272309ec1e0c: Status 404 returned error can't find the container with id db21ad1175c5bf7afffe6a8a269ab6c438cdbba805d8ba22a642272309ec1e0c Aug 04 11:05:32.000489 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:32.000456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" event={"ID":"74965014-c6d0-4178-b6ef-55e068bc8506","Type":"ContainerStarted","Data":"f9b4e0091d8afbb64a1f571ed3454661c46c375b3961cbfe6d2fd1838b943c69"} Aug 04 11:05:32.000870 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:32.000517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" event={"ID":"74965014-c6d0-4178-b6ef-55e068bc8506","Type":"ContainerStarted","Data":"db21ad1175c5bf7afffe6a8a269ab6c438cdbba805d8ba22a642272309ec1e0c"} Aug 04 11:05:32.015475 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:32.015413 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" podStartSLOduration=2.015391798 podStartE2EDuration="2.015391798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:05:32.014633751 +0000 UTC m=+2292.966041301" watchObservedRunningTime="2026-08-04 11:05:32.015391798 +0000 UTC m=+2292.966799346" Aug 04 11:05:33.006617 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:33.006575 2577 generic.go:356] "Generic (PLEG): container finished" podID="74965014-c6d0-4178-b6ef-55e068bc8506" containerID="f9b4e0091d8afbb64a1f571ed3454661c46c375b3961cbfe6d2fd1838b943c69" exitCode=0 Aug 04 11:05:33.007031 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:33.006660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" event={"ID":"74965014-c6d0-4178-b6ef-55e068bc8506","Type":"ContainerDied","Data":"f9b4e0091d8afbb64a1f571ed3454661c46c375b3961cbfe6d2fd1838b943c69"} Aug 04 11:05:34.139537 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:34.139512 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" Aug 04 11:05:34.188362 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:34.188328 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74965014-c6d0-4178-b6ef-55e068bc8506-maas-api-service-ca\") pod \"74965014-c6d0-4178-b6ef-55e068bc8506\" (UID: \"74965014-c6d0-4178-b6ef-55e068bc8506\") " Aug 04 11:05:34.188766 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:34.188741 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74965014-c6d0-4178-b6ef-55e068bc8506-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74965014-c6d0-4178-b6ef-55e068bc8506" (UID: "74965014-c6d0-4178-b6ef-55e068bc8506"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:05:34.289320 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:34.289256 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74965014-c6d0-4178-b6ef-55e068bc8506-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:05:35.017897 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:35.017860 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" event={"ID":"74965014-c6d0-4178-b6ef-55e068bc8506","Type":"ContainerDied","Data":"db21ad1175c5bf7afffe6a8a269ab6c438cdbba805d8ba22a642272309ec1e0c"} Aug 04 11:05:35.017897 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:35.017886 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr" Aug 04 11:05:35.018100 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:35.017913 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db21ad1175c5bf7afffe6a8a269ab6c438cdbba805d8ba22a642272309ec1e0c" Aug 04 11:05:40.712068 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:40.712031 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf"] Aug 04 11:05:40.712553 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:40.712305 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" podUID="c8f77950-dee2-4c1f-a6a9-5d15949093f0" containerName="maas-api" containerID="cri-o://474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42" gracePeriod=30 Aug 04 11:05:40.933175 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:40.933144 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr"] Aug 04 11:05:40.937178 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:40.937154 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a18af8fb-mvpcr"] Aug 04 11:05:40.957428 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:40.957408 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:05:41.052607 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.052579 2577 generic.go:356] "Generic (PLEG): container finished" podID="c8f77950-dee2-4c1f-a6a9-5d15949093f0" containerID="474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42" exitCode=0 Aug 04 11:05:41.052751 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.052618 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" Aug 04 11:05:41.052751 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.052689 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" event={"ID":"c8f77950-dee2-4c1f-a6a9-5d15949093f0","Type":"ContainerDied","Data":"474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42"} Aug 04 11:05:41.052751 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.052737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf" event={"ID":"c8f77950-dee2-4c1f-a6a9-5d15949093f0","Type":"ContainerDied","Data":"728dbfde5955807c4cd0084276ca397273638f12b1ea7c6c6c6d0e8e0c589cdd"} Aug 04 11:05:41.052894 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.052763 2577 scope.go:117] "RemoveContainer" containerID="474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42" Aug 04 11:05:41.056878 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.056855 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls\") pod \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " Aug 04 11:05:41.056986 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.056892 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f77950-dee2-4c1f-a6a9-5d15949093f0-openshift-service-ca\") pod \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " Aug 04 11:05:41.057106 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.057078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbw42\" (UniqueName: \"kubernetes.io/projected/c8f77950-dee2-4c1f-a6a9-5d15949093f0-kube-api-access-dbw42\") pod \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\" (UID: \"c8f77950-dee2-4c1f-a6a9-5d15949093f0\") " Aug 04 11:05:41.057297 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.057278 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f77950-dee2-4c1f-a6a9-5d15949093f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8f77950-dee2-4c1f-a6a9-5d15949093f0" (UID: "c8f77950-dee2-4c1f-a6a9-5d15949093f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:05:41.057436 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.057421 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f77950-dee2-4c1f-a6a9-5d15949093f0-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:05:41.059150 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.059127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8f77950-dee2-4c1f-a6a9-5d15949093f0" (UID: "c8f77950-dee2-4c1f-a6a9-5d15949093f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:05:41.059385 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.059358 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f77950-dee2-4c1f-a6a9-5d15949093f0-kube-api-access-dbw42" (OuterVolumeSpecName: "kube-api-access-dbw42") pod "c8f77950-dee2-4c1f-a6a9-5d15949093f0" (UID: "c8f77950-dee2-4c1f-a6a9-5d15949093f0"). InnerVolumeSpecName "kube-api-access-dbw42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:05:41.064114 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.064096 2577 scope.go:117] "RemoveContainer" containerID="474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42" Aug 04 11:05:41.064375 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:05:41.064355 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42\": container with ID starting with 474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42 not found: ID does not exist" containerID="474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42" Aug 04 11:05:41.064438 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.064384 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42"} err="failed to get container status \"474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42\": rpc error: code = NotFound desc = could not find container \"474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42\": container with ID starting with 474938462ca38be76fbb83d09d283b05a6ef574f57f764ee38976664805afa42 not found: ID does not exist" Aug 04 11:05:41.158229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.158201 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f77950-dee2-4c1f-a6a9-5d15949093f0-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:05:41.158229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.158226 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbw42\" (UniqueName: \"kubernetes.io/projected/c8f77950-dee2-4c1f-a6a9-5d15949093f0-kube-api-access-dbw42\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:05:41.375063 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.375036 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf"] Aug 04 11:05:41.378843 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.378808 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a18af8fb-795948b5c-f6sqf"] Aug 04 11:05:41.532420 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.532386 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74965014-c6d0-4178-b6ef-55e068bc8506" path="/var/lib/kubelet/pods/74965014-c6d0-4178-b6ef-55e068bc8506/volumes" Aug 04 11:05:41.532787 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:05:41.532773 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f77950-dee2-4c1f-a6a9-5d15949093f0" path="/var/lib/kubelet/pods/c8f77950-dee2-4c1f-a6a9-5d15949093f0/volumes" Aug 04 11:07:19.764604 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:19.764577 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:07:19.774901 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:19.774881 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:07:43.161967 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.161878 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf"] Aug 04 11:07:43.162509 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.162492 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f77950-dee2-4c1f-a6a9-5d15949093f0" containerName="maas-api" Aug 04 11:07:43.162552 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.162513 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f77950-dee2-4c1f-a6a9-5d15949093f0" containerName="maas-api" Aug 04 11:07:43.162552 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.162528 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74965014-c6d0-4178-b6ef-55e068bc8506" containerName="revoke-keys" Aug 04 11:07:43.162552 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.162537 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74965014-c6d0-4178-b6ef-55e068bc8506" containerName="revoke-keys" Aug 04 11:07:43.163469 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.163442 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f77950-dee2-4c1f-a6a9-5d15949093f0" containerName="maas-api" Aug 04 11:07:43.163645 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.163632 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74965014-c6d0-4178-b6ef-55e068bc8506" containerName="revoke-keys" Aug 04 11:07:43.169889 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.169866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.172506 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.172468 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-78f47f-serving-cert\"" Aug 04 11:07:43.175637 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.175608 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf"] Aug 04 11:07:43.226587 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.226549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6mk\" (UniqueName: \"kubernetes.io/projected/18dbfc05-787a-4fde-b3c2-2fae826dbd40-kube-api-access-dj6mk\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.226772 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.226689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18dbfc05-787a-4fde-b3c2-2fae826dbd40-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.226772 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.226761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.328058 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.328018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18dbfc05-787a-4fde-b3c2-2fae826dbd40-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.328270 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.328117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.328270 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.328142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6mk\" (UniqueName: \"kubernetes.io/projected/18dbfc05-787a-4fde-b3c2-2fae826dbd40-kube-api-access-dj6mk\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.328393 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:07:43.328269 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-serving-cert: secret "maas-api-e2e-inf-a-78f47f-serving-cert" not found Aug 04 11:07:43.328393 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:07:43.328357 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls podName:18dbfc05-787a-4fde-b3c2-2fae826dbd40 nodeName:}" failed. No retries permitted until 2026-08-04 11:07:43.82833661 +0000 UTC m=+2424.779744149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls") pod "maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" (UID: "18dbfc05-787a-4fde-b3c2-2fae826dbd40") : secret "maas-api-e2e-inf-a-78f47f-serving-cert" not found Aug 04 11:07:43.328982 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.328963 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18dbfc05-787a-4fde-b3c2-2fae826dbd40-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.336374 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.336350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6mk\" (UniqueName: \"kubernetes.io/projected/18dbfc05-787a-4fde-b3c2-2fae826dbd40-kube-api-access-dj6mk\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.833246 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.833205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:43.835697 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:43.835669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls\") pod \"maas-api-e2e-inf-a-78f47f-6ff865d466-427gf\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:44.086006 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:44.085907 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:44.242029 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:44.242002 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf"] Aug 04 11:07:44.243518 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:07:44.243491 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dbfc05_787a_4fde_b3c2_2fae826dbd40.slice/crio-0c96ab71c0e4d5d36a7f8a0200ae0788432ab57eaf9bc273792f8849c4e0b6f8 WatchSource:0}: Error finding container 0c96ab71c0e4d5d36a7f8a0200ae0788432ab57eaf9bc273792f8849c4e0b6f8: Status 404 returned error can't find the container with id 0c96ab71c0e4d5d36a7f8a0200ae0788432ab57eaf9bc273792f8849c4e0b6f8 Aug 04 11:07:44.719035 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:44.719005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" event={"ID":"18dbfc05-787a-4fde-b3c2-2fae826dbd40","Type":"ContainerStarted","Data":"0c96ab71c0e4d5d36a7f8a0200ae0788432ab57eaf9bc273792f8849c4e0b6f8"} Aug 04 11:07:45.727522 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:45.727484 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" event={"ID":"18dbfc05-787a-4fde-b3c2-2fae826dbd40","Type":"ContainerStarted","Data":"bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea"} Aug 04 11:07:45.727967 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:45.727557 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:07:45.744507 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:45.744453 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" podStartSLOduration=2.338662388 podStartE2EDuration="2.744434493s" totalImagesPullingTime="405.772105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:07:45.743179418 +0000 UTC m=+2426.694586967" watchObservedRunningTime="2026-08-04 11:07:45.744434493 +0000 UTC m=+2426.695842041" Aug 04 11:07:51.739268 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:07:51.739234 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:08:12.202261 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.202218 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk"] Aug 04 11:08:12.205864 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.205847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.208134 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.208107 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1c7c26-serving-cert\"" Aug 04 11:08:12.214562 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.214526 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk"] Aug 04 11:08:12.302003 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.301966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.302184 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.302056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.302184 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.302101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm2wm\" (UniqueName: \"kubernetes.io/projected/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-kube-api-access-mm2wm\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.402706 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.402671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.402950 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.402719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm2wm\" (UniqueName: \"kubernetes.io/projected/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-kube-api-access-mm2wm\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.402950 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.402797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.403342 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.403318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.405325 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.405298 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.410908 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.410883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm2wm\" (UniqueName: \"kubernetes.io/projected/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-kube-api-access-mm2wm\") pod \"maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.518367 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.518325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:12.662044 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.659484 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk"] Aug 04 11:08:12.882537 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:12.882456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" event={"ID":"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c","Type":"ContainerStarted","Data":"876ced5752d14c9c1c16b425485b6f765b72b8875f4ef2ce2280084c8550832f"} Aug 04 11:08:13.890621 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:13.890581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" event={"ID":"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c","Type":"ContainerStarted","Data":"e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a"} Aug 04 11:08:13.891037 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:13.890680 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:08:13.906823 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:13.906768 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" podStartSLOduration=1.3959617340000001 podStartE2EDuration="1.90675286s" totalImagesPullingTime="510.791126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:08:13.905197297 +0000 UTC m=+2454.856604846" watchObservedRunningTime="2026-08-04 11:08:13.90675286 +0000 UTC m=+2454.858160408" Aug 04 11:08:19.893495 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:19.893462 2577 scope.go:117] "RemoveContainer" containerID="256bf958da45c0cd2715090500c6cbe478bb4932c3ee10ece519b0224c5c6576" Aug 04 11:08:19.901882 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:08:19.901856 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:10:02.078805 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.078768 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg"] Aug 04 11:10:02.082365 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.082347 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" Aug 04 11:10:02.084089 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.084070 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 11:10:02.089563 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.089532 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg"] Aug 04 11:10:02.176077 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.176036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e5ca5f-525b-45e8-9aec-f120c10afdf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg\" (UID: \"98e5ca5f-525b-45e8-9aec-f120c10afdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" Aug 04 11:10:02.276546 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.276509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e5ca5f-525b-45e8-9aec-f120c10afdf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg\" (UID: \"98e5ca5f-525b-45e8-9aec-f120c10afdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" Aug 04 11:10:02.277180 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.277160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e5ca5f-525b-45e8-9aec-f120c10afdf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg\" (UID: \"98e5ca5f-525b-45e8-9aec-f120c10afdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" Aug 04 11:10:02.394794 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.394709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" Aug 04 11:10:02.529485 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:02.529460 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg"] Aug 04 11:10:02.531311 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:10:02.531283 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e5ca5f_525b_45e8_9aec_f120c10afdf9.slice/crio-bd07c8362222b28e3100cfe50225d18644682c13b978e87fe9da833fda6932e2 WatchSource:0}: Error finding container bd07c8362222b28e3100cfe50225d18644682c13b978e87fe9da833fda6932e2: Status 404 returned error can't find the container with id bd07c8362222b28e3100cfe50225d18644682c13b978e87fe9da833fda6932e2 Aug 04 11:10:03.494755 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:03.494670 2577 generic.go:356] "Generic (PLEG): container finished" podID="98e5ca5f-525b-45e8-9aec-f120c10afdf9" containerID="66e416977214d222a708d5afacb13c5cc1ad1199c6189492c123be8433a0789c" exitCode=0 Aug 04 11:10:03.495132 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:03.494756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" event={"ID":"98e5ca5f-525b-45e8-9aec-f120c10afdf9","Type":"ContainerDied","Data":"66e416977214d222a708d5afacb13c5cc1ad1199c6189492c123be8433a0789c"} Aug 04 11:10:03.495132 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:03.494813 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" event={"ID":"98e5ca5f-525b-45e8-9aec-f120c10afdf9","Type":"ContainerStarted","Data":"bd07c8362222b28e3100cfe50225d18644682c13b978e87fe9da833fda6932e2"} Aug 04 11:10:04.630240 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:04.630215 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" Aug 04 11:10:04.701608 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:04.701574 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e5ca5f-525b-45e8-9aec-f120c10afdf9-maas-api-service-ca\") pod \"98e5ca5f-525b-45e8-9aec-f120c10afdf9\" (UID: \"98e5ca5f-525b-45e8-9aec-f120c10afdf9\") " Aug 04 11:10:04.702026 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:04.702004 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e5ca5f-525b-45e8-9aec-f120c10afdf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98e5ca5f-525b-45e8-9aec-f120c10afdf9" (UID: "98e5ca5f-525b-45e8-9aec-f120c10afdf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:04.802792 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:04.802703 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e5ca5f-525b-45e8-9aec-f120c10afdf9-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:05.509281 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:05.509231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" event={"ID":"98e5ca5f-525b-45e8-9aec-f120c10afdf9","Type":"ContainerDied","Data":"bd07c8362222b28e3100cfe50225d18644682c13b978e87fe9da833fda6932e2"} Aug 04 11:10:05.509488 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:05.509296 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd07c8362222b28e3100cfe50225d18644682c13b978e87fe9da833fda6932e2" Aug 04 11:10:05.509488 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:05.509359 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg" Aug 04 11:10:06.533624 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.533591 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf"] Aug 04 11:10:06.534070 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.533801 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" podUID="18dbfc05-787a-4fde-b3c2-2fae826dbd40" containerName="maas-api" containerID="cri-o://bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea" gracePeriod=30 Aug 04 11:10:06.725125 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.725089 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg"] Aug 04 11:10:06.727619 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.727591 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78f47f-z6wsg"] Aug 04 11:10:06.790652 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.790596 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:10:06.923768 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.923735 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls\") pod \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " Aug 04 11:10:06.923980 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.923783 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj6mk\" (UniqueName: \"kubernetes.io/projected/18dbfc05-787a-4fde-b3c2-2fae826dbd40-kube-api-access-dj6mk\") pod \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " Aug 04 11:10:06.923980 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.923812 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18dbfc05-787a-4fde-b3c2-2fae826dbd40-openshift-service-ca\") pod \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\" (UID: \"18dbfc05-787a-4fde-b3c2-2fae826dbd40\") " Aug 04 11:10:06.924196 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.924170 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18dbfc05-787a-4fde-b3c2-2fae826dbd40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18dbfc05-787a-4fde-b3c2-2fae826dbd40" (UID: "18dbfc05-787a-4fde-b3c2-2fae826dbd40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:06.925888 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.925855 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18dbfc05-787a-4fde-b3c2-2fae826dbd40" (UID: "18dbfc05-787a-4fde-b3c2-2fae826dbd40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:10:06.925888 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:06.925859 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18dbfc05-787a-4fde-b3c2-2fae826dbd40-kube-api-access-dj6mk" (OuterVolumeSpecName: "kube-api-access-dj6mk") pod "18dbfc05-787a-4fde-b3c2-2fae826dbd40" (UID: "18dbfc05-787a-4fde-b3c2-2fae826dbd40"). InnerVolumeSpecName "kube-api-access-dj6mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:07.024897 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.024861 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18dbfc05-787a-4fde-b3c2-2fae826dbd40-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:07.024897 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.024893 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj6mk\" (UniqueName: \"kubernetes.io/projected/18dbfc05-787a-4fde-b3c2-2fae826dbd40-kube-api-access-dj6mk\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:07.025103 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.024906 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18dbfc05-787a-4fde-b3c2-2fae826dbd40-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:07.521755 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.521719 2577 generic.go:356] "Generic (PLEG): container finished" podID="18dbfc05-787a-4fde-b3c2-2fae826dbd40" containerID="bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea" exitCode=0 Aug 04 11:10:07.521956 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.521765 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" Aug 04 11:10:07.521956 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.521799 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" event={"ID":"18dbfc05-787a-4fde-b3c2-2fae826dbd40","Type":"ContainerDied","Data":"bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea"} Aug 04 11:10:07.521956 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.521875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" event={"ID":"18dbfc05-787a-4fde-b3c2-2fae826dbd40","Type":"ContainerDied","Data":"0c96ab71c0e4d5d36a7f8a0200ae0788432ab57eaf9bc273792f8849c4e0b6f8"} Aug 04 11:10:07.521956 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.521910 2577 scope.go:117] "RemoveContainer" containerID="bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea" Aug 04 11:10:07.532564 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.532547 2577 scope.go:117] "RemoveContainer" containerID="bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea" Aug 04 11:10:07.532886 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:10:07.532862 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea\": container with ID starting with bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea not found: ID does not exist" containerID="bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea" Aug 04 11:10:07.532989 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.532893 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea"} err="failed to get container status \"bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea\": rpc error: code = NotFound desc = could not find container \"bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea\": container with ID starting with bc40ebcc78b941a5aad0120b921b0a9b49881528161c6c2cbe24010f5e9f81ea not found: ID does not exist" Aug 04 11:10:07.533076 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.533053 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98e5ca5f-525b-45e8-9aec-f120c10afdf9" path="/var/lib/kubelet/pods/98e5ca5f-525b-45e8-9aec-f120c10afdf9/volumes" Aug 04 11:10:07.543312 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.543283 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf"] Aug 04 11:10:07.547298 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:07.547271 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf"] Aug 04 11:10:09.532531 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:09.532497 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18dbfc05-787a-4fde-b3c2-2fae826dbd40" path="/var/lib/kubelet/pods/18dbfc05-787a-4fde-b3c2-2fae826dbd40/volumes" Aug 04 11:10:09.734813 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:09.734770 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78f47f-6ff865d466-427gf" podUID="18dbfc05-787a-4fde-b3c2-2fae826dbd40" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.62:8443/health\": context deadline exceeded" Aug 04 11:10:19.093080 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.093045 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv"] Aug 04 11:10:19.093453 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.093441 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98e5ca5f-525b-45e8-9aec-f120c10afdf9" containerName="revoke-keys" Aug 04 11:10:19.093511 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.093455 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98e5ca5f-525b-45e8-9aec-f120c10afdf9" containerName="revoke-keys" Aug 04 11:10:19.093511 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.093477 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18dbfc05-787a-4fde-b3c2-2fae826dbd40" containerName="maas-api" Aug 04 11:10:19.093511 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.093482 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18dbfc05-787a-4fde-b3c2-2fae826dbd40" containerName="maas-api" Aug 04 11:10:19.093607 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.093552 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18dbfc05-787a-4fde-b3c2-2fae826dbd40" containerName="maas-api" Aug 04 11:10:19.093607 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.093567 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98e5ca5f-525b-45e8-9aec-f120c10afdf9" containerName="revoke-keys" Aug 04 11:10:19.098009 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.097990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" Aug 04 11:10:19.100682 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.100651 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 11:10:19.103505 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.103481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv"] Aug 04 11:10:19.233068 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.233026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f742fa-afdd-485d-b6af-6731901f23a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv\" (UID: \"74f742fa-afdd-485d-b6af-6731901f23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" Aug 04 11:10:19.334188 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.334148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f742fa-afdd-485d-b6af-6731901f23a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv\" (UID: \"74f742fa-afdd-485d-b6af-6731901f23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" Aug 04 11:10:19.334759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.334741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f742fa-afdd-485d-b6af-6731901f23a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv\" (UID: \"74f742fa-afdd-485d-b6af-6731901f23a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" Aug 04 11:10:19.409942 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.409877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" Aug 04 11:10:19.543223 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.543199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv"] Aug 04 11:10:19.545093 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:10:19.545068 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f742fa_afdd_485d_b6af_6731901f23a0.slice/crio-7eafec2d5ba80adc765664cabfe7ca3e4cc0debb9cf339c76a198bfc89ad0f1f WatchSource:0}: Error finding container 7eafec2d5ba80adc765664cabfe7ca3e4cc0debb9cf339c76a198bfc89ad0f1f: Status 404 returned error can't find the container with id 7eafec2d5ba80adc765664cabfe7ca3e4cc0debb9cf339c76a198bfc89ad0f1f Aug 04 11:10:19.594234 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:19.594212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" event={"ID":"74f742fa-afdd-485d-b6af-6731901f23a0","Type":"ContainerStarted","Data":"7eafec2d5ba80adc765664cabfe7ca3e4cc0debb9cf339c76a198bfc89ad0f1f"} Aug 04 11:10:20.600924 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:20.600889 2577 generic.go:356] "Generic (PLEG): container finished" podID="74f742fa-afdd-485d-b6af-6731901f23a0" containerID="4eeeafee64d314519d21323412063032033abfe3ef8ab9dd86112ad8c1270517" exitCode=0 Aug 04 11:10:20.601323 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:20.600951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" event={"ID":"74f742fa-afdd-485d-b6af-6731901f23a0","Type":"ContainerDied","Data":"4eeeafee64d314519d21323412063032033abfe3ef8ab9dd86112ad8c1270517"} Aug 04 11:10:21.739598 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:21.739572 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" Aug 04 11:10:21.855243 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:21.855206 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f742fa-afdd-485d-b6af-6731901f23a0-maas-api-service-ca\") pod \"74f742fa-afdd-485d-b6af-6731901f23a0\" (UID: \"74f742fa-afdd-485d-b6af-6731901f23a0\") " Aug 04 11:10:21.855570 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:21.855545 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f742fa-afdd-485d-b6af-6731901f23a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74f742fa-afdd-485d-b6af-6731901f23a0" (UID: "74f742fa-afdd-485d-b6af-6731901f23a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:21.956386 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:21.956305 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f742fa-afdd-485d-b6af-6731901f23a0-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:22.613093 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:22.613066 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" Aug 04 11:10:22.613262 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:22.613061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv" event={"ID":"74f742fa-afdd-485d-b6af-6731901f23a0","Type":"ContainerDied","Data":"7eafec2d5ba80adc765664cabfe7ca3e4cc0debb9cf339c76a198bfc89ad0f1f"} Aug 04 11:10:22.613262 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:22.613191 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eafec2d5ba80adc765664cabfe7ca3e4cc0debb9cf339c76a198bfc89ad0f1f" Aug 04 11:10:29.554383 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.554347 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk"] Aug 04 11:10:29.554796 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.554586 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" podUID="7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" containerName="maas-api" containerID="cri-o://e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a" gracePeriod=30 Aug 04 11:10:29.723497 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.723464 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv"] Aug 04 11:10:29.727114 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.727084 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1c7c26-qdfdv"] Aug 04 11:10:29.796340 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.796317 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:10:29.930022 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.929951 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-openshift-service-ca\") pod \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " Aug 04 11:10:29.930161 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.930033 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm2wm\" (UniqueName: \"kubernetes.io/projected/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-kube-api-access-mm2wm\") pod \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " Aug 04 11:10:29.930161 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.930107 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-maas-api-tls\") pod \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\" (UID: \"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c\") " Aug 04 11:10:29.930292 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.930271 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" (UID: "7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:10:29.932047 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.932025 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" (UID: "7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:10:29.932168 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:29.932141 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-kube-api-access-mm2wm" (OuterVolumeSpecName: "kube-api-access-mm2wm") pod "7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" (UID: "7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c"). InnerVolumeSpecName "kube-api-access-mm2wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:10:30.031095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.031064 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm2wm\" (UniqueName: \"kubernetes.io/projected/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-kube-api-access-mm2wm\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:30.031095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.031090 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:30.031095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.031101 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:10:30.657694 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.657663 2577 generic.go:356] "Generic (PLEG): container finished" podID="7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" containerID="e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a" exitCode=0 Aug 04 11:10:30.658124 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.657700 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" Aug 04 11:10:30.658124 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.657748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" event={"ID":"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c","Type":"ContainerDied","Data":"e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a"} Aug 04 11:10:30.658124 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.657795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk" event={"ID":"7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c","Type":"ContainerDied","Data":"876ced5752d14c9c1c16b425485b6f765b72b8875f4ef2ce2280084c8550832f"} Aug 04 11:10:30.658124 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.657816 2577 scope.go:117] "RemoveContainer" containerID="e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a" Aug 04 11:10:30.667619 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.667602 2577 scope.go:117] "RemoveContainer" containerID="e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a" Aug 04 11:10:30.667904 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:10:30.667886 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a\": container with ID starting with e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a not found: ID does not exist" containerID="e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a" Aug 04 11:10:30.667980 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.667911 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a"} err="failed to get container status \"e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a\": rpc error: code = NotFound desc = could not find container \"e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a\": container with ID starting with e7a7f1c6f1a516797ef6f4357d7205ab571f3a495fc9e2d06be024fdddcb284a not found: ID does not exist" Aug 04 11:10:30.684273 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.684249 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk"] Aug 04 11:10:30.687602 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:30.687585 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c7c26-d8cc9db7f-785kk"] Aug 04 11:10:31.532219 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:31.532188 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f742fa-afdd-485d-b6af-6731901f23a0" path="/var/lib/kubelet/pods/74f742fa-afdd-485d-b6af-6731901f23a0/volumes" Aug 04 11:10:31.532581 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:31.532566 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" path="/var/lib/kubelet/pods/7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c/volumes" Aug 04 11:10:41.989920 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.989882 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m"] Aug 04 11:10:41.990511 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.990463 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f742fa-afdd-485d-b6af-6731901f23a0" containerName="revoke-keys" Aug 04 11:10:41.990511 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.990479 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f742fa-afdd-485d-b6af-6731901f23a0" containerName="revoke-keys" Aug 04 11:10:41.990511 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.990505 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" containerName="maas-api" Aug 04 11:10:41.990693 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.990514 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" containerName="maas-api" Aug 04 11:10:41.990693 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.990617 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f742fa-afdd-485d-b6af-6731901f23a0" containerName="revoke-keys" Aug 04 11:10:41.990693 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.990636 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c723bf2-df8b-4a5d-97c9-ae4d69b94a6c" containerName="maas-api" Aug 04 11:10:41.995575 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.995551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:41.998697 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:41.997553 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1154a6-serving-cert\"" Aug 04 11:10:42.000988 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.000964 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m"] Aug 04 11:10:42.037980 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.037951 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbdn\" (UniqueName: \"kubernetes.io/projected/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-kube-api-access-8mbdn\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.038093 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.038018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-openshift-service-ca\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.038140 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.038107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.139013 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.138937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-openshift-service-ca\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.139013 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.139003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.139200 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.139050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbdn\" (UniqueName: \"kubernetes.io/projected/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-kube-api-access-8mbdn\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.139200 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:10:42.139160 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-serving-cert: secret "maas-api-e2e-api-a-1154a6-serving-cert" not found Aug 04 11:10:42.139272 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:10:42.139229 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls podName:e55bfb48-d374-4fd1-a5c1-f4643ec30cae nodeName:}" failed. No retries permitted until 2026-08-04 11:10:42.63921448 +0000 UTC m=+2603.590622011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls") pod "maas-api-e2e-api-a-1154a6-99495f48c-6v47m" (UID: "e55bfb48-d374-4fd1-a5c1-f4643ec30cae") : secret "maas-api-e2e-api-a-1154a6-serving-cert" not found Aug 04 11:10:42.139681 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.139661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-openshift-service-ca\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.149182 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.149153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbdn\" (UniqueName: \"kubernetes.io/projected/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-kube-api-access-8mbdn\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.642127 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.642093 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.644469 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.644436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls\") pod \"maas-api-e2e-api-a-1154a6-99495f48c-6v47m\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:42.907928 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:42.907819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:43.042566 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:43.042541 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m"] Aug 04 11:10:43.045157 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:10:43.045133 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55bfb48_d374_4fd1_a5c1_f4643ec30cae.slice/crio-1badfedb1348ef16769c3db0fa03101391c08e3b38b824ff4287487604845895 WatchSource:0}: Error finding container 1badfedb1348ef16769c3db0fa03101391c08e3b38b824ff4287487604845895: Status 404 returned error can't find the container with id 1badfedb1348ef16769c3db0fa03101391c08e3b38b824ff4287487604845895 Aug 04 11:10:43.046421 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:43.046404 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:10:43.730107 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:43.730067 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" event={"ID":"e55bfb48-d374-4fd1-a5c1-f4643ec30cae","Type":"ContainerStarted","Data":"19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf"} Aug 04 11:10:43.730264 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:43.730133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" event={"ID":"e55bfb48-d374-4fd1-a5c1-f4643ec30cae","Type":"ContainerStarted","Data":"1badfedb1348ef16769c3db0fa03101391c08e3b38b824ff4287487604845895"} Aug 04 11:10:43.730314 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:43.730279 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:10:43.747078 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:43.747034 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" podStartSLOduration=2.261162414 podStartE2EDuration="2.747021311s" totalImagesPullingTime="485.858897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:10:43.745421928 +0000 UTC m=+2604.696829478" watchObservedRunningTime="2026-08-04 11:10:43.747021311 +0000 UTC m=+2604.698428859" Aug 04 11:10:49.741685 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:10:49.741653 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:11:11.110723 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.110689 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8"] Aug 04 11:11:11.114245 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.114230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.115986 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.115966 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-74c752-serving-cert\"" Aug 04 11:11:11.125261 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.125238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8"] Aug 04 11:11:11.191204 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.191179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4js\" (UniqueName: \"kubernetes.io/projected/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-kube-api-access-lq4js\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.191340 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.191209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-openshift-service-ca\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.191340 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.191238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.291858 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.291816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4js\" (UniqueName: \"kubernetes.io/projected/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-kube-api-access-lq4js\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.292007 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.291941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-openshift-service-ca\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.292007 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.291996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.292142 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:11:11.292128 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-serving-cert: secret "maas-api-e2e-api-b-74c752-serving-cert" not found Aug 04 11:11:11.292196 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:11:11.292187 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls podName:b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8 nodeName:}" failed. No retries permitted until 2026-08-04 11:11:11.792170895 +0000 UTC m=+2632.743578422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls") pod "maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" (UID: "b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8") : secret "maas-api-e2e-api-b-74c752-serving-cert" not found Aug 04 11:11:11.292550 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.292533 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-openshift-service-ca\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.299940 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.299925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4js\" (UniqueName: \"kubernetes.io/projected/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-kube-api-access-lq4js\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.795465 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.795429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:11.798156 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:11.798128 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls\") pod \"maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:12.025961 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:12.025931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:12.157435 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:12.157324 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8"] Aug 04 11:11:12.160509 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:11:12.160482 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33b07a6_a77f_43f8_9f95_6f1b2bbea7e8.slice/crio-dda07b415c72d8afd75c027440a14d8c610c21690bd27088d4f394bde6a5e092 WatchSource:0}: Error finding container dda07b415c72d8afd75c027440a14d8c610c21690bd27088d4f394bde6a5e092: Status 404 returned error can't find the container with id dda07b415c72d8afd75c027440a14d8c610c21690bd27088d4f394bde6a5e092 Aug 04 11:11:12.891251 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:12.891162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" event={"ID":"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8","Type":"ContainerStarted","Data":"d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408"} Aug 04 11:11:12.891251 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:12.891223 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" event={"ID":"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8","Type":"ContainerStarted","Data":"dda07b415c72d8afd75c027440a14d8c610c21690bd27088d4f394bde6a5e092"} Aug 04 11:11:12.891482 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:12.891255 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:12.910135 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:12.910088 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" podStartSLOduration=1.5312195229999999 podStartE2EDuration="1.910074689s" totalImagesPullingTime="378.855166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:11:12.907542441 +0000 UTC m=+2633.858949989" watchObservedRunningTime="2026-08-04 11:11:12.910074689 +0000 UTC m=+2633.861482237" Aug 04 11:11:18.902188 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:18.902156 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:11:58.685565 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:58.685529 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m"] Aug 04 11:11:58.686124 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:58.685757 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" podUID="e55bfb48-d374-4fd1-a5c1-f4643ec30cae" containerName="maas-api" containerID="cri-o://19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf" gracePeriod=30 Aug 04 11:11:58.938414 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:58.938348 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:11:59.101345 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.101310 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-openshift-service-ca\") pod \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " Aug 04 11:11:59.101531 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.101361 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls\") pod \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " Aug 04 11:11:59.101531 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.101395 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mbdn\" (UniqueName: \"kubernetes.io/projected/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-kube-api-access-8mbdn\") pod \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\" (UID: \"e55bfb48-d374-4fd1-a5c1-f4643ec30cae\") " Aug 04 11:11:59.101661 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.101584 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e55bfb48-d374-4fd1-a5c1-f4643ec30cae" (UID: "e55bfb48-d374-4fd1-a5c1-f4643ec30cae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:11:59.103531 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.103502 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e55bfb48-d374-4fd1-a5c1-f4643ec30cae" (UID: "e55bfb48-d374-4fd1-a5c1-f4643ec30cae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:11:59.103638 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.103570 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-kube-api-access-8mbdn" (OuterVolumeSpecName: "kube-api-access-8mbdn") pod "e55bfb48-d374-4fd1-a5c1-f4643ec30cae" (UID: "e55bfb48-d374-4fd1-a5c1-f4643ec30cae"). InnerVolumeSpecName "kube-api-access-8mbdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:11:59.167877 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.167845 2577 generic.go:356] "Generic (PLEG): container finished" podID="e55bfb48-d374-4fd1-a5c1-f4643ec30cae" containerID="19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf" exitCode=0 Aug 04 11:11:59.168014 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.167883 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" Aug 04 11:11:59.168014 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.167918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" event={"ID":"e55bfb48-d374-4fd1-a5c1-f4643ec30cae","Type":"ContainerDied","Data":"19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf"} Aug 04 11:11:59.168014 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.167973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m" event={"ID":"e55bfb48-d374-4fd1-a5c1-f4643ec30cae","Type":"ContainerDied","Data":"1badfedb1348ef16769c3db0fa03101391c08e3b38b824ff4287487604845895"} Aug 04 11:11:59.168014 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.167995 2577 scope.go:117] "RemoveContainer" containerID="19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf" Aug 04 11:11:59.177506 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.177490 2577 scope.go:117] "RemoveContainer" containerID="19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf" Aug 04 11:11:59.177734 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:11:59.177716 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf\": container with ID starting with 19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf not found: ID does not exist" containerID="19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf" Aug 04 11:11:59.177771 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.177745 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf"} err="failed to get container status \"19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf\": rpc error: code = NotFound desc = could not find container \"19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf\": container with ID starting with 19c970e31de48ece26af55218ba8335cb953ac9e03c3926053337f892043aaaf not found: ID does not exist" Aug 04 11:11:59.187747 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.187727 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m"] Aug 04 11:11:59.193645 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.193598 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1154a6-99495f48c-6v47m"] Aug 04 11:11:59.202486 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.202467 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mbdn\" (UniqueName: \"kubernetes.io/projected/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-kube-api-access-8mbdn\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:11:59.202486 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.202486 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:11:59.202609 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.202497 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55bfb48-d374-4fd1-a5c1-f4643ec30cae-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:11:59.532576 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:11:59.532543 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e55bfb48-d374-4fd1-a5c1-f4643ec30cae" path="/var/lib/kubelet/pods/e55bfb48-d374-4fd1-a5c1-f4643ec30cae/volumes" Aug 04 11:12:05.964596 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:05.964562 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt"] Aug 04 11:12:05.965087 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:05.964987 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55bfb48-d374-4fd1-a5c1-f4643ec30cae" containerName="maas-api" Aug 04 11:12:05.965087 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:05.965000 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55bfb48-d374-4fd1-a5c1-f4643ec30cae" containerName="maas-api" Aug 04 11:12:05.965087 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:05.965073 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55bfb48-d374-4fd1-a5c1-f4643ec30cae" containerName="maas-api" Aug 04 11:12:05.969376 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:05.969359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" Aug 04 11:12:05.971434 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:05.971415 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qptbt\"" Aug 04 11:12:05.976902 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:05.976876 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt"] Aug 04 11:12:06.067820 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:06.067788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef56f47-6504-46ce-8004-c12787b7f12c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74c752-x8rmt\" (UID: \"1ef56f47-6504-46ce-8004-c12787b7f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" Aug 04 11:12:06.168691 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:06.168654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef56f47-6504-46ce-8004-c12787b7f12c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74c752-x8rmt\" (UID: \"1ef56f47-6504-46ce-8004-c12787b7f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" Aug 04 11:12:06.169350 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:06.169330 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef56f47-6504-46ce-8004-c12787b7f12c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74c752-x8rmt\" (UID: \"1ef56f47-6504-46ce-8004-c12787b7f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" Aug 04 11:12:06.281649 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:06.281620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" Aug 04 11:12:06.414961 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:06.414935 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt"] Aug 04 11:12:06.417097 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:12:06.417071 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef56f47_6504_46ce_8004_c12787b7f12c.slice/crio-a6f5f58ca6ddfa64795876b4ab3198cfb261ae8686dc9ed66d30b773f3a48377 WatchSource:0}: Error finding container a6f5f58ca6ddfa64795876b4ab3198cfb261ae8686dc9ed66d30b773f3a48377: Status 404 returned error can't find the container with id a6f5f58ca6ddfa64795876b4ab3198cfb261ae8686dc9ed66d30b773f3a48377 Aug 04 11:12:07.214994 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:07.214961 2577 generic.go:356] "Generic (PLEG): container finished" podID="1ef56f47-6504-46ce-8004-c12787b7f12c" containerID="abeabe6fe73097847938a8466380b6c018416d2b123e8ab2c4d90d9e3d4de0a9" exitCode=0 Aug 04 11:12:07.215384 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:07.215005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" event={"ID":"1ef56f47-6504-46ce-8004-c12787b7f12c","Type":"ContainerDied","Data":"abeabe6fe73097847938a8466380b6c018416d2b123e8ab2c4d90d9e3d4de0a9"} Aug 04 11:12:07.215384 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:07.215049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" event={"ID":"1ef56f47-6504-46ce-8004-c12787b7f12c","Type":"ContainerStarted","Data":"a6f5f58ca6ddfa64795876b4ab3198cfb261ae8686dc9ed66d30b773f3a48377"} Aug 04 11:12:08.349206 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:08.349179 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" Aug 04 11:12:08.492887 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:08.492783 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef56f47-6504-46ce-8004-c12787b7f12c-maas-api-service-ca\") pod \"1ef56f47-6504-46ce-8004-c12787b7f12c\" (UID: \"1ef56f47-6504-46ce-8004-c12787b7f12c\") " Aug 04 11:12:08.493111 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:08.493089 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef56f47-6504-46ce-8004-c12787b7f12c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ef56f47-6504-46ce-8004-c12787b7f12c" (UID: "1ef56f47-6504-46ce-8004-c12787b7f12c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:12:08.594087 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:08.594062 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef56f47-6504-46ce-8004-c12787b7f12c-maas-api-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:12:09.226699 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:09.226667 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" Aug 04 11:12:09.226890 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:09.226712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt" event={"ID":"1ef56f47-6504-46ce-8004-c12787b7f12c","Type":"ContainerDied","Data":"a6f5f58ca6ddfa64795876b4ab3198cfb261ae8686dc9ed66d30b773f3a48377"} Aug 04 11:12:09.226890 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:09.226758 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f5f58ca6ddfa64795876b4ab3198cfb261ae8686dc9ed66d30b773f3a48377" Aug 04 11:12:19.802473 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:19.802449 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:12:19.813663 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:19.813640 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:12:19.952651 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:19.952619 2577 scope.go:117] "RemoveContainer" containerID="f9b4e0091d8afbb64a1f571ed3454661c46c375b3961cbfe6d2fd1838b943c69" Aug 04 11:12:21.508785 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.508756 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8"] Aug 04 11:12:21.509182 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.509012 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" podUID="b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" containerName="maas-api" containerID="cri-o://d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408" gracePeriod=30 Aug 04 11:12:21.737849 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.737798 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt"] Aug 04 11:12:21.740132 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.740106 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74c752-x8rmt"] Aug 04 11:12:21.756611 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.756587 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:12:21.809067 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.809005 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq4js\" (UniqueName: \"kubernetes.io/projected/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-kube-api-access-lq4js\") pod \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " Aug 04 11:12:21.809067 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.809044 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-openshift-service-ca\") pod \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " Aug 04 11:12:21.809246 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.809079 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls\") pod \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\" (UID: \"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8\") " Aug 04 11:12:21.809410 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.809386 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" (UID: "b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:12:21.810983 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.810963 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" (UID: "b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:12:21.811131 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.811114 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-kube-api-access-lq4js" (OuterVolumeSpecName: "kube-api-access-lq4js") pod "b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" (UID: "b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8"). InnerVolumeSpecName "kube-api-access-lq4js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:12:21.910770 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.910739 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq4js\" (UniqueName: \"kubernetes.io/projected/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-kube-api-access-lq4js\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:12:21.910770 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.910765 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:12:21.910770 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:21.910775 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:12:22.301888 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.301856 2577 generic.go:356] "Generic (PLEG): container finished" podID="b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" containerID="d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408" exitCode=0 Aug 04 11:12:22.302057 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.301900 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" Aug 04 11:12:22.302057 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.301930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" event={"ID":"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8","Type":"ContainerDied","Data":"d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408"} Aug 04 11:12:22.302057 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.301981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8" event={"ID":"b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8","Type":"ContainerDied","Data":"dda07b415c72d8afd75c027440a14d8c610c21690bd27088d4f394bde6a5e092"} Aug 04 11:12:22.302057 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.302004 2577 scope.go:117] "RemoveContainer" containerID="d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408" Aug 04 11:12:22.311637 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.311619 2577 scope.go:117] "RemoveContainer" containerID="d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408" Aug 04 11:12:22.311882 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:12:22.311859 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408\": container with ID starting with d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408 not found: ID does not exist" containerID="d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408" Aug 04 11:12:22.311938 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.311891 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408"} err="failed to get container status \"d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408\": rpc error: code = NotFound desc = could not find container \"d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408\": container with ID starting with d4dd0b55fde36ea026603c91f06e055702575ed32952d8a1720466b433b84408 not found: ID does not exist" Aug 04 11:12:22.326424 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.326403 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8"] Aug 04 11:12:22.328676 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:22.328657 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-74c752-69fc5468cd-cwcv8"] Aug 04 11:12:23.531947 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:23.531915 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef56f47-6504-46ce-8004-c12787b7f12c" path="/var/lib/kubelet/pods/1ef56f47-6504-46ce-8004-c12787b7f12c/volumes" Aug 04 11:12:23.532322 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:23.532297 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" path="/var/lib/kubelet/pods/b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8/volumes" Aug 04 11:12:33.608413 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.608379 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg"] Aug 04 11:12:33.608805 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.608768 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" containerName="maas-api" Aug 04 11:12:33.608805 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.608783 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" containerName="maas-api" Aug 04 11:12:33.608805 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.608796 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef56f47-6504-46ce-8004-c12787b7f12c" containerName="revoke-keys" Aug 04 11:12:33.608805 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.608802 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef56f47-6504-46ce-8004-c12787b7f12c" containerName="revoke-keys" Aug 04 11:12:33.608965 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.608912 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33b07a6-a77f-43f8-9f95-6f1b2bbea7e8" containerName="maas-api" Aug 04 11:12:33.608965 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.608920 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef56f47-6504-46ce-8004-c12787b7f12c" containerName="revoke-keys" Aug 04 11:12:33.612107 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.612084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.614317 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.614295 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-bc4a78-serving-cert\"" Aug 04 11:12:33.621523 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.621498 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg"] Aug 04 11:12:33.712494 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.712463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.712676 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.712541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7tps\" (UniqueName: \"kubernetes.io/projected/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-kube-api-access-z7tps\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.712676 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.712622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.813658 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.813621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.813823 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.813674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7tps\" (UniqueName: \"kubernetes.io/projected/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-kube-api-access-z7tps\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.813823 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.813702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.814357 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.814328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.816247 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.816219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.821992 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.821965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7tps\" (UniqueName: \"kubernetes.io/projected/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-kube-api-access-z7tps\") pod \"maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:33.924009 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:33.923933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:34.053381 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:34.053357 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg"] Aug 04 11:12:34.055510 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:12:34.055486 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a4d994_3e68_4e60_8a3f_e5db386b17f8.slice/crio-22109639c8d577febc2b5bd24aa83af34ebbd1af871314d21804122d0dd21acf WatchSource:0}: Error finding container 22109639c8d577febc2b5bd24aa83af34ebbd1af871314d21804122d0dd21acf: Status 404 returned error can't find the container with id 22109639c8d577febc2b5bd24aa83af34ebbd1af871314d21804122d0dd21acf Aug 04 11:12:34.372753 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:34.372703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" event={"ID":"f9a4d994-3e68-4e60-8a3f-e5db386b17f8","Type":"ContainerStarted","Data":"22109639c8d577febc2b5bd24aa83af34ebbd1af871314d21804122d0dd21acf"} Aug 04 11:12:35.379193 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:35.379161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" event={"ID":"f9a4d994-3e68-4e60-8a3f-e5db386b17f8","Type":"ContainerStarted","Data":"4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3"} Aug 04 11:12:35.379607 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:35.379292 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:12:35.395396 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:35.395352 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" podStartSLOduration=2.012147172 podStartE2EDuration="2.395338913s" totalImagesPullingTime="383.191741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:12:35.394172912 +0000 UTC m=+2716.345580461" watchObservedRunningTime="2026-08-04 11:12:35.395338913 +0000 UTC m=+2716.346746460" Aug 04 11:12:41.390349 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:12:41.390317 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:13:01.698231 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.698195 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt"] Aug 04 11:13:01.701772 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.701756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.703552 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.703533 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-460cac-serving-cert\"" Aug 04 11:13:01.710866 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.710841 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt"] Aug 04 11:13:01.874242 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.874205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4767125a-878e-4490-b3d2-69eb14ca063d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.874242 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.874243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwbx\" (UniqueName: \"kubernetes.io/projected/4767125a-878e-4490-b3d2-69eb14ca063d-kube-api-access-qrwbx\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.874458 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.874270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.975413 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.975307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4767125a-878e-4490-b3d2-69eb14ca063d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.975413 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.975357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwbx\" (UniqueName: \"kubernetes.io/projected/4767125a-878e-4490-b3d2-69eb14ca063d-kube-api-access-qrwbx\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.975413 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.975379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.975703 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:13:01.975468 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-serving-cert: secret "maas-api-e2e-shared-b-460cac-serving-cert" not found Aug 04 11:13:01.975703 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:13:01.975527 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls podName:4767125a-878e-4490-b3d2-69eb14ca063d nodeName:}" failed. No retries permitted until 2026-08-04 11:13:02.475511961 +0000 UTC m=+2743.426919489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls") pod "maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" (UID: "4767125a-878e-4490-b3d2-69eb14ca063d") : secret "maas-api-e2e-shared-b-460cac-serving-cert" not found Aug 04 11:13:01.976104 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.976083 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4767125a-878e-4490-b3d2-69eb14ca063d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:01.983906 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:01.983882 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwbx\" (UniqueName: \"kubernetes.io/projected/4767125a-878e-4490-b3d2-69eb14ca063d-kube-api-access-qrwbx\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:02.480386 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:02.480343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:02.482658 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:02.482640 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls\") pod \"maas-api-e2e-shared-b-460cac-5759cb594d-8txzt\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:02.613974 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:02.613938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:02.747098 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:02.747015 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt"] Aug 04 11:13:02.749986 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:13:02.749961 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4767125a_878e_4490_b3d2_69eb14ca063d.slice/crio-d97885215b1ecb3fbe2af723e53eea4691d50c565b9c317f0751e34e71bc5dd6 WatchSource:0}: Error finding container d97885215b1ecb3fbe2af723e53eea4691d50c565b9c317f0751e34e71bc5dd6: Status 404 returned error can't find the container with id d97885215b1ecb3fbe2af723e53eea4691d50c565b9c317f0751e34e71bc5dd6 Aug 04 11:13:03.536964 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:03.536932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" event={"ID":"4767125a-878e-4490-b3d2-69eb14ca063d","Type":"ContainerStarted","Data":"ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037"} Aug 04 11:13:03.537144 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:03.536982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" event={"ID":"4767125a-878e-4490-b3d2-69eb14ca063d","Type":"ContainerStarted","Data":"d97885215b1ecb3fbe2af723e53eea4691d50c565b9c317f0751e34e71bc5dd6"} Aug 04 11:13:03.537144 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:03.537030 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:13:03.554045 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:03.553993 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" podStartSLOduration=2.199006576 podStartE2EDuration="2.553976277s" totalImagesPullingTime="354.969701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:13:03.552726466 +0000 UTC m=+2744.504134014" watchObservedRunningTime="2026-08-04 11:13:03.553976277 +0000 UTC m=+2744.505383825" Aug 04 11:13:09.548328 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:13:09.548298 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:16:20.008493 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:16:20.008448 2577 scope.go:117] "RemoveContainer" containerID="4eeeafee64d314519d21323412063032033abfe3ef8ab9dd86112ad8c1270517" Aug 04 11:16:20.018267 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:16:20.018244 2577 scope.go:117] "RemoveContainer" containerID="66e416977214d222a708d5afacb13c5cc1ad1199c6189492c123be8433a0789c" Aug 04 11:17:19.839734 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:17:19.839696 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:17:19.852278 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:17:19.852253 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:18:20.055148 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:18:20.055104 2577 scope.go:117] "RemoveContainer" containerID="abeabe6fe73097847938a8466380b6c018416d2b123e8ab2c4d90d9e3d4de0a9" Aug 04 11:20:47.562740 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.562707 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7625-140190814333744"] Aug 04 11:20:47.566266 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.566248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:47.568217 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.568194 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6n6vh\"" Aug 04 11:20:47.574666 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.574085 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814333744"] Aug 04 11:20:47.621778 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.621748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg65m\" (UniqueName: \"kubernetes.io/projected/a76aa4e8-da5c-401a-be7d-69dccec7c2f1-kube-api-access-kg65m\") pod \"test-curl-7625-140190814333744\" (UID: \"a76aa4e8-da5c-401a-be7d-69dccec7c2f1\") " pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:47.722960 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.722927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg65m\" (UniqueName: \"kubernetes.io/projected/a76aa4e8-da5c-401a-be7d-69dccec7c2f1-kube-api-access-kg65m\") pod \"test-curl-7625-140190814333744\" (UID: \"a76aa4e8-da5c-401a-be7d-69dccec7c2f1\") " pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:47.732091 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.732066 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg65m\" (UniqueName: \"kubernetes.io/projected/a76aa4e8-da5c-401a-be7d-69dccec7c2f1-kube-api-access-kg65m\") pod \"test-curl-7625-140190814333744\" (UID: \"a76aa4e8-da5c-401a-be7d-69dccec7c2f1\") " pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:47.877531 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:47.877428 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:48.014429 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:48.014389 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814333744"] Aug 04 11:20:48.019393 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:20:48.019364 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda76aa4e8_da5c_401a_be7d_69dccec7c2f1.slice/crio-81f46b30da12a283873e4951f2ec0a9e098dc4c2f1c086b350d68033ca619425 WatchSource:0}: Error finding container 81f46b30da12a283873e4951f2ec0a9e098dc4c2f1c086b350d68033ca619425: Status 404 returned error can't find the container with id 81f46b30da12a283873e4951f2ec0a9e098dc4c2f1c086b350d68033ca619425 Aug 04 11:20:48.020703 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:48.020687 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 11:20:48.089538 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:48.089496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814333744" event={"ID":"a76aa4e8-da5c-401a-be7d-69dccec7c2f1","Type":"ContainerStarted","Data":"81f46b30da12a283873e4951f2ec0a9e098dc4c2f1c086b350d68033ca619425"} Aug 04 11:20:49.096515 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:49.096480 2577 generic.go:356] "Generic (PLEG): container finished" podID="a76aa4e8-da5c-401a-be7d-69dccec7c2f1" containerID="6ebd2a20bba66918c22d3932e4d0fe525cda2e2f25bbb9e93cb24d79ad218046" exitCode=0 Aug 04 11:20:49.096947 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:49.096582 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814333744" event={"ID":"a76aa4e8-da5c-401a-be7d-69dccec7c2f1","Type":"ContainerDied","Data":"6ebd2a20bba66918c22d3932e4d0fe525cda2e2f25bbb9e93cb24d79ad218046"} Aug 04 11:20:50.226608 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.226578 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:50.248248 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.248210 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg65m\" (UniqueName: \"kubernetes.io/projected/a76aa4e8-da5c-401a-be7d-69dccec7c2f1-kube-api-access-kg65m\") pod \"a76aa4e8-da5c-401a-be7d-69dccec7c2f1\" (UID: \"a76aa4e8-da5c-401a-be7d-69dccec7c2f1\") " Aug 04 11:20:50.250909 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.250874 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76aa4e8-da5c-401a-be7d-69dccec7c2f1-kube-api-access-kg65m" (OuterVolumeSpecName: "kube-api-access-kg65m") pod "a76aa4e8-da5c-401a-be7d-69dccec7c2f1" (UID: "a76aa4e8-da5c-401a-be7d-69dccec7c2f1"). InnerVolumeSpecName "kube-api-access-kg65m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:20:50.273496 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.273473 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7625-140190814333744_a76aa4e8-da5c-401a-be7d-69dccec7c2f1/test-curl-7625-140190814333744/0.log" Aug 04 11:20:50.298849 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.298798 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7625-140190814333744"] Aug 04 11:20:50.301514 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.301494 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7625-140190814333744"] Aug 04 11:20:50.349665 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.349641 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg65m\" (UniqueName: \"kubernetes.io/projected/a76aa4e8-da5c-401a-be7d-69dccec7c2f1-kube-api-access-kg65m\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:20:50.437520 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.437462 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7625-140190814331440"] Aug 04 11:20:50.437919 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.437905 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76aa4e8-da5c-401a-be7d-69dccec7c2f1" containerName="test-curl-7625-140190814333744" Aug 04 11:20:50.437968 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.437921 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76aa4e8-da5c-401a-be7d-69dccec7c2f1" containerName="test-curl-7625-140190814333744" Aug 04 11:20:50.438002 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.437992 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76aa4e8-da5c-401a-be7d-69dccec7c2f1" containerName="test-curl-7625-140190814333744" Aug 04 11:20:50.441076 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.441059 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814331440" Aug 04 11:20:50.449674 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.449655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814331440"] Aug 04 11:20:50.551546 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.551511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2qs\" (UniqueName: \"kubernetes.io/projected/8fbd8495-9dfd-4863-8872-2118c9a4835c-kube-api-access-gf2qs\") pod \"test-curl-7625-140190814331440\" (UID: \"8fbd8495-9dfd-4863-8872-2118c9a4835c\") " pod="opendatahub/test-curl-7625-140190814331440" Aug 04 11:20:50.652327 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.652285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2qs\" (UniqueName: \"kubernetes.io/projected/8fbd8495-9dfd-4863-8872-2118c9a4835c-kube-api-access-gf2qs\") pod \"test-curl-7625-140190814331440\" (UID: \"8fbd8495-9dfd-4863-8872-2118c9a4835c\") " pod="opendatahub/test-curl-7625-140190814331440" Aug 04 11:20:50.661371 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.661348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2qs\" (UniqueName: \"kubernetes.io/projected/8fbd8495-9dfd-4863-8872-2118c9a4835c-kube-api-access-gf2qs\") pod \"test-curl-7625-140190814331440\" (UID: \"8fbd8495-9dfd-4863-8872-2118c9a4835c\") " pod="opendatahub/test-curl-7625-140190814331440" Aug 04 11:20:50.751511 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.751489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814331440" Aug 04 11:20:50.883389 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:50.883357 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814331440"] Aug 04 11:20:50.885227 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:20:50.885200 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fbd8495_9dfd_4863_8872_2118c9a4835c.slice/crio-16cd031cdf41d3ab8fb0dd1b94380eeba145743ca599e3d298d4b74039d4ed95 WatchSource:0}: Error finding container 16cd031cdf41d3ab8fb0dd1b94380eeba145743ca599e3d298d4b74039d4ed95: Status 404 returned error can't find the container with id 16cd031cdf41d3ab8fb0dd1b94380eeba145743ca599e3d298d4b74039d4ed95 Aug 04 11:20:51.108568 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:51.108484 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:51.108568 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:51.108500 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81f46b30da12a283873e4951f2ec0a9e098dc4c2f1c086b350d68033ca619425" Aug 04 11:20:51.109911 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:51.109879 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814331440" event={"ID":"8fbd8495-9dfd-4863-8872-2118c9a4835c","Type":"ContainerStarted","Data":"16cd031cdf41d3ab8fb0dd1b94380eeba145743ca599e3d298d4b74039d4ed95"} Aug 04 11:20:51.119239 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:20:51.119209 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7625-140190814333744\" is forbidden: User \"system:node:ip-10-0-139-198.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-198.ec2.internal' and this object" podUID="a76aa4e8-da5c-401a-be7d-69dccec7c2f1" pod="opendatahub/test-curl-7625-140190814333744" Aug 04 11:20:51.533232 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:51.533196 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76aa4e8-da5c-401a-be7d-69dccec7c2f1" path="/var/lib/kubelet/pods/a76aa4e8-da5c-401a-be7d-69dccec7c2f1/volumes" Aug 04 11:20:52.117020 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:52.116981 2577 generic.go:356] "Generic (PLEG): container finished" podID="8fbd8495-9dfd-4863-8872-2118c9a4835c" containerID="c5920274636f3675bfb5c9c685b9d4f4622f04361ee2a22cff54648bd73eac8e" exitCode=0 Aug 04 11:20:52.117190 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:52.117068 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814331440" event={"ID":"8fbd8495-9dfd-4863-8872-2118c9a4835c","Type":"ContainerDied","Data":"c5920274636f3675bfb5c9c685b9d4f4622f04361ee2a22cff54648bd73eac8e"} Aug 04 11:20:53.259624 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.259601 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814331440" Aug 04 11:20:53.280845 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.280808 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7625-140190814331440_8fbd8495-9dfd-4863-8872-2118c9a4835c/test-curl-7625-140190814331440/0.log" Aug 04 11:20:53.307041 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.307015 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7625-140190814331440"] Aug 04 11:20:53.311194 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.311171 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7625-140190814331440"] Aug 04 11:20:53.380707 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.380633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf2qs\" (UniqueName: \"kubernetes.io/projected/8fbd8495-9dfd-4863-8872-2118c9a4835c-kube-api-access-gf2qs\") pod \"8fbd8495-9dfd-4863-8872-2118c9a4835c\" (UID: \"8fbd8495-9dfd-4863-8872-2118c9a4835c\") " Aug 04 11:20:53.382603 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.382581 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbd8495-9dfd-4863-8872-2118c9a4835c-kube-api-access-gf2qs" (OuterVolumeSpecName: "kube-api-access-gf2qs") pod "8fbd8495-9dfd-4863-8872-2118c9a4835c" (UID: "8fbd8495-9dfd-4863-8872-2118c9a4835c"). InnerVolumeSpecName "kube-api-access-gf2qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:20:53.481639 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.481600 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2qs\" (UniqueName: \"kubernetes.io/projected/8fbd8495-9dfd-4863-8872-2118c9a4835c-kube-api-access-gf2qs\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:20:53.493569 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.493539 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:53.493953 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.493940 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbd8495-9dfd-4863-8872-2118c9a4835c" containerName="test-curl-7625-140190814331440" Aug 04 11:20:53.494006 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.493955 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbd8495-9dfd-4863-8872-2118c9a4835c" containerName="test-curl-7625-140190814331440" Aug 04 11:20:53.494041 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.494031 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbd8495-9dfd-4863-8872-2118c9a4835c" containerName="test-curl-7625-140190814331440" Aug 04 11:20:53.497022 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.497007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:53.504774 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.504750 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:53.533064 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.533034 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fbd8495-9dfd-4863-8872-2118c9a4835c" path="/var/lib/kubelet/pods/8fbd8495-9dfd-4863-8872-2118c9a4835c/volumes" Aug 04 11:20:53.582407 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.582368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rw4\" (UniqueName: \"kubernetes.io/projected/aa782d05-0fa2-4464-96f0-cb882e37cbc4-kube-api-access-f2rw4\") pod \"test-curl-7625-140190814440736\" (UID: \"aa782d05-0fa2-4464-96f0-cb882e37cbc4\") " pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:53.683895 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.683800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rw4\" (UniqueName: \"kubernetes.io/projected/aa782d05-0fa2-4464-96f0-cb882e37cbc4-kube-api-access-f2rw4\") pod \"test-curl-7625-140190814440736\" (UID: \"aa782d05-0fa2-4464-96f0-cb882e37cbc4\") " pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:53.698229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.698195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rw4\" (UniqueName: \"kubernetes.io/projected/aa782d05-0fa2-4464-96f0-cb882e37cbc4-kube-api-access-f2rw4\") pod \"test-curl-7625-140190814440736\" (UID: \"aa782d05-0fa2-4464-96f0-cb882e37cbc4\") " pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:53.807749 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.807713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:53.939118 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:53.939095 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:53.941275 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:20:53.941250 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa782d05_0fa2_4464_96f0_cb882e37cbc4.slice/crio-9695ed8e1b7042d92290bdcccf56e290c3b33abd74ca17c4d9bee2d059b512af WatchSource:0}: Error finding container 9695ed8e1b7042d92290bdcccf56e290c3b33abd74ca17c4d9bee2d059b512af: Status 404 returned error can't find the container with id 9695ed8e1b7042d92290bdcccf56e290c3b33abd74ca17c4d9bee2d059b512af Aug 04 11:20:54.129190 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:54.129164 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814331440" Aug 04 11:20:54.129355 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:54.129207 2577 scope.go:117] "RemoveContainer" containerID="c5920274636f3675bfb5c9c685b9d4f4622f04361ee2a22cff54648bd73eac8e" Aug 04 11:20:54.130486 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:54.130401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814440736" event={"ID":"aa782d05-0fa2-4464-96f0-cb882e37cbc4","Type":"ContainerStarted","Data":"9695ed8e1b7042d92290bdcccf56e290c3b33abd74ca17c4d9bee2d059b512af"} Aug 04 11:20:55.139154 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:55.139119 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa782d05-0fa2-4464-96f0-cb882e37cbc4" containerID="c878056afb9bcb44d6ff13c2612338d377c27b15681089e50f961bb2c95ae593" exitCode=0 Aug 04 11:20:55.139558 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:55.139189 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814440736" event={"ID":"aa782d05-0fa2-4464-96f0-cb882e37cbc4","Type":"ContainerDied","Data":"c878056afb9bcb44d6ff13c2612338d377c27b15681089e50f961bb2c95ae593"} Aug 04 11:20:56.274791 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.274764 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:56.298991 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.298960 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7625-140190814440736_aa782d05-0fa2-4464-96f0-cb882e37cbc4/test-curl-7625-140190814440736/0.log" Aug 04 11:20:56.310110 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.310087 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2rw4\" (UniqueName: \"kubernetes.io/projected/aa782d05-0fa2-4464-96f0-cb882e37cbc4-kube-api-access-f2rw4\") pod \"aa782d05-0fa2-4464-96f0-cb882e37cbc4\" (UID: \"aa782d05-0fa2-4464-96f0-cb882e37cbc4\") " Aug 04 11:20:56.312127 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.312104 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa782d05-0fa2-4464-96f0-cb882e37cbc4-kube-api-access-f2rw4" (OuterVolumeSpecName: "kube-api-access-f2rw4") pod "aa782d05-0fa2-4464-96f0-cb882e37cbc4" (UID: "aa782d05-0fa2-4464-96f0-cb882e37cbc4"). InnerVolumeSpecName "kube-api-access-f2rw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:20:56.327265 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.327243 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:56.331440 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.331420 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:56.411102 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.411051 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2rw4\" (UniqueName: \"kubernetes.io/projected/aa782d05-0fa2-4464-96f0-cb882e37cbc4-kube-api-access-f2rw4\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:20:56.449613 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.449587 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:56.450006 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.449992 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa782d05-0fa2-4464-96f0-cb882e37cbc4" containerName="test-curl-7625-140190814440736" Aug 04 11:20:56.450062 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.450008 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa782d05-0fa2-4464-96f0-cb882e37cbc4" containerName="test-curl-7625-140190814440736" Aug 04 11:20:56.450106 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.450065 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa782d05-0fa2-4464-96f0-cb882e37cbc4" containerName="test-curl-7625-140190814440736" Aug 04 11:20:56.453038 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.453025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:56.461449 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.461427 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:56.512262 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.512227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh65x\" (UniqueName: \"kubernetes.io/projected/2d962f14-9be1-488a-bf13-85f70578343a-kube-api-access-jh65x\") pod \"test-curl-7625-140190814440736\" (UID: \"2d962f14-9be1-488a-bf13-85f70578343a\") " pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:56.613346 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.613309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh65x\" (UniqueName: \"kubernetes.io/projected/2d962f14-9be1-488a-bf13-85f70578343a-kube-api-access-jh65x\") pod \"test-curl-7625-140190814440736\" (UID: \"2d962f14-9be1-488a-bf13-85f70578343a\") " pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:56.622558 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.622526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh65x\" (UniqueName: \"kubernetes.io/projected/2d962f14-9be1-488a-bf13-85f70578343a-kube-api-access-jh65x\") pod \"test-curl-7625-140190814440736\" (UID: \"2d962f14-9be1-488a-bf13-85f70578343a\") " pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:56.763030 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.763006 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:56.900924 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:56.900898 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:56.903504 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:20:56.903483 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d962f14_9be1_488a_bf13_85f70578343a.slice/crio-e55bee3e1a60dd75ceca9eff9997148a7a1629380982cfd974f68168b8c123fb WatchSource:0}: Error finding container e55bee3e1a60dd75ceca9eff9997148a7a1629380982cfd974f68168b8c123fb: Status 404 returned error can't find the container with id e55bee3e1a60dd75ceca9eff9997148a7a1629380982cfd974f68168b8c123fb Aug 04 11:20:57.151652 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:57.151574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814440736" event={"ID":"2d962f14-9be1-488a-bf13-85f70578343a","Type":"ContainerStarted","Data":"e55bee3e1a60dd75ceca9eff9997148a7a1629380982cfd974f68168b8c123fb"} Aug 04 11:20:57.152765 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:57.152730 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9695ed8e1b7042d92290bdcccf56e290c3b33abd74ca17c4d9bee2d059b512af" Aug 04 11:20:57.152919 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:57.152766 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:57.164951 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:57.164919 2577 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7625-140190814440736" oldPodUID="aa782d05-0fa2-4464-96f0-cb882e37cbc4" podUID="2d962f14-9be1-488a-bf13-85f70578343a" Aug 04 11:20:57.532759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:57.532727 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa782d05-0fa2-4464-96f0-cb882e37cbc4" path="/var/lib/kubelet/pods/aa782d05-0fa2-4464-96f0-cb882e37cbc4/volumes" Aug 04 11:20:58.160137 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:58.160101 2577 generic.go:356] "Generic (PLEG): container finished" podID="2d962f14-9be1-488a-bf13-85f70578343a" containerID="7d6eebdf1bf9a63a65d8f7d17db45e83745e18be0ec1614e2ee1bfa14dcc7ba6" exitCode=0 Aug 04 11:20:58.160330 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:58.160187 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814440736" event={"ID":"2d962f14-9be1-488a-bf13-85f70578343a","Type":"ContainerDied","Data":"7d6eebdf1bf9a63a65d8f7d17db45e83745e18be0ec1614e2ee1bfa14dcc7ba6"} Aug 04 11:20:59.298663 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.298637 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:20:59.317675 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.317643 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7625-140190814440736_2d962f14-9be1-488a-bf13-85f70578343a/test-curl-7625-140190814440736/0.log" Aug 04 11:20:59.342997 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.342966 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:59.345304 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.345282 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7625-140190814440736"] Aug 04 11:20:59.440964 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.440896 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh65x\" (UniqueName: \"kubernetes.io/projected/2d962f14-9be1-488a-bf13-85f70578343a-kube-api-access-jh65x\") pod \"2d962f14-9be1-488a-bf13-85f70578343a\" (UID: \"2d962f14-9be1-488a-bf13-85f70578343a\") " Aug 04 11:20:59.442934 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.442909 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d962f14-9be1-488a-bf13-85f70578343a-kube-api-access-jh65x" (OuterVolumeSpecName: "kube-api-access-jh65x") pod "2d962f14-9be1-488a-bf13-85f70578343a" (UID: "2d962f14-9be1-488a-bf13-85f70578343a"). InnerVolumeSpecName "kube-api-access-jh65x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:20:59.486847 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.486805 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:20:59.487220 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.487206 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d962f14-9be1-488a-bf13-85f70578343a" containerName="test-curl-7625-140190814440736" Aug 04 11:20:59.487269 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.487222 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d962f14-9be1-488a-bf13-85f70578343a" containerName="test-curl-7625-140190814440736" Aug 04 11:20:59.487324 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.487314 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d962f14-9be1-488a-bf13-85f70578343a" containerName="test-curl-7625-140190814440736" Aug 04 11:20:59.490310 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.490294 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:20:59.498122 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.498100 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:20:59.532705 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.532678 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d962f14-9be1-488a-bf13-85f70578343a" path="/var/lib/kubelet/pods/2d962f14-9be1-488a-bf13-85f70578343a/volumes" Aug 04 11:20:59.541906 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.541886 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch28w\" (UniqueName: \"kubernetes.io/projected/51f79b08-9a6b-4009-8bf2-d020f5b05683-kube-api-access-ch28w\") pod \"test-curl-7625-140190814294352\" (UID: \"51f79b08-9a6b-4009-8bf2-d020f5b05683\") " pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:20:59.542025 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.542008 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh65x\" (UniqueName: \"kubernetes.io/projected/2d962f14-9be1-488a-bf13-85f70578343a-kube-api-access-jh65x\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:20:59.642748 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.642695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch28w\" (UniqueName: \"kubernetes.io/projected/51f79b08-9a6b-4009-8bf2-d020f5b05683-kube-api-access-ch28w\") pod \"test-curl-7625-140190814294352\" (UID: \"51f79b08-9a6b-4009-8bf2-d020f5b05683\") " pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:20:59.650950 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.650928 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch28w\" (UniqueName: \"kubernetes.io/projected/51f79b08-9a6b-4009-8bf2-d020f5b05683-kube-api-access-ch28w\") pod \"test-curl-7625-140190814294352\" (UID: \"51f79b08-9a6b-4009-8bf2-d020f5b05683\") " pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:20:59.799858 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.799806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:20:59.931007 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:20:59.930983 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:20:59.933077 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:20:59.933051 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f79b08_9a6b_4009_8bf2_d020f5b05683.slice/crio-bd679a259cc9228246638a18df73c9ef18a34b88f04abf9e76eea593578adb87 WatchSource:0}: Error finding container bd679a259cc9228246638a18df73c9ef18a34b88f04abf9e76eea593578adb87: Status 404 returned error can't find the container with id bd679a259cc9228246638a18df73c9ef18a34b88f04abf9e76eea593578adb87 Aug 04 11:21:00.174263 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:00.174183 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814440736" Aug 04 11:21:00.174263 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:00.174227 2577 scope.go:117] "RemoveContainer" containerID="7d6eebdf1bf9a63a65d8f7d17db45e83745e18be0ec1614e2ee1bfa14dcc7ba6" Aug 04 11:21:00.175548 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:00.175520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814294352" event={"ID":"51f79b08-9a6b-4009-8bf2-d020f5b05683","Type":"ContainerStarted","Data":"bd679a259cc9228246638a18df73c9ef18a34b88f04abf9e76eea593578adb87"} Aug 04 11:21:01.182026 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:01.181993 2577 generic.go:356] "Generic (PLEG): container finished" podID="51f79b08-9a6b-4009-8bf2-d020f5b05683" containerID="9b8e1979622e07fae89cfe1ee8a3b98c12dc713b21e446fc26c361b9451a871f" exitCode=0 Aug 04 11:21:01.182478 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:01.182075 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814294352" event={"ID":"51f79b08-9a6b-4009-8bf2-d020f5b05683","Type":"ContainerDied","Data":"9b8e1979622e07fae89cfe1ee8a3b98c12dc713b21e446fc26c361b9451a871f"} Aug 04 11:21:02.323212 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.323188 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:02.346880 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.346851 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7625-140190814294352_51f79b08-9a6b-4009-8bf2-d020f5b05683/test-curl-7625-140190814294352/0.log" Aug 04 11:21:02.375082 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.375053 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:21:02.378872 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.378849 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:21:02.469319 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.469236 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch28w\" (UniqueName: \"kubernetes.io/projected/51f79b08-9a6b-4009-8bf2-d020f5b05683-kube-api-access-ch28w\") pod \"51f79b08-9a6b-4009-8bf2-d020f5b05683\" (UID: \"51f79b08-9a6b-4009-8bf2-d020f5b05683\") " Aug 04 11:21:02.471279 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.471244 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f79b08-9a6b-4009-8bf2-d020f5b05683-kube-api-access-ch28w" (OuterVolumeSpecName: "kube-api-access-ch28w") pod "51f79b08-9a6b-4009-8bf2-d020f5b05683" (UID: "51f79b08-9a6b-4009-8bf2-d020f5b05683"). InnerVolumeSpecName "kube-api-access-ch28w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:21:02.508795 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.508754 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:21:02.509680 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.509653 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f79b08-9a6b-4009-8bf2-d020f5b05683" containerName="test-curl-7625-140190814294352" Aug 04 11:21:02.509680 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.509680 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f79b08-9a6b-4009-8bf2-d020f5b05683" containerName="test-curl-7625-140190814294352" Aug 04 11:21:02.509937 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.509922 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f79b08-9a6b-4009-8bf2-d020f5b05683" containerName="test-curl-7625-140190814294352" Aug 04 11:21:02.514146 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.514125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:02.517560 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.517533 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:21:02.570570 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.570532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gq7\" (UniqueName: \"kubernetes.io/projected/6055f39d-9ddf-4561-8d4f-4beb32a2802c-kube-api-access-d7gq7\") pod \"test-curl-7625-140190814294352\" (UID: \"6055f39d-9ddf-4561-8d4f-4beb32a2802c\") " pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:02.570736 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.570610 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch28w\" (UniqueName: \"kubernetes.io/projected/51f79b08-9a6b-4009-8bf2-d020f5b05683-kube-api-access-ch28w\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:21:02.671463 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.671425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gq7\" (UniqueName: \"kubernetes.io/projected/6055f39d-9ddf-4561-8d4f-4beb32a2802c-kube-api-access-d7gq7\") pod \"test-curl-7625-140190814294352\" (UID: \"6055f39d-9ddf-4561-8d4f-4beb32a2802c\") " pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:02.679441 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.679415 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gq7\" (UniqueName: \"kubernetes.io/projected/6055f39d-9ddf-4561-8d4f-4beb32a2802c-kube-api-access-d7gq7\") pod \"test-curl-7625-140190814294352\" (UID: \"6055f39d-9ddf-4561-8d4f-4beb32a2802c\") " pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:02.825937 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.825901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:02.964081 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:02.964053 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:21:02.965985 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:21:02.965963 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6055f39d_9ddf_4561_8d4f_4beb32a2802c.slice/crio-fe50aa9454180b1699d8d77c5f41693c04c5357743c13ca236ed8524aedc4b22 WatchSource:0}: Error finding container fe50aa9454180b1699d8d77c5f41693c04c5357743c13ca236ed8524aedc4b22: Status 404 returned error can't find the container with id fe50aa9454180b1699d8d77c5f41693c04c5357743c13ca236ed8524aedc4b22 Aug 04 11:21:03.196171 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:03.196091 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd679a259cc9228246638a18df73c9ef18a34b88f04abf9e76eea593578adb87" Aug 04 11:21:03.196171 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:03.196097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:03.197227 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:03.197201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814294352" event={"ID":"6055f39d-9ddf-4561-8d4f-4beb32a2802c","Type":"ContainerStarted","Data":"fe50aa9454180b1699d8d77c5f41693c04c5357743c13ca236ed8524aedc4b22"} Aug 04 11:21:03.208411 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:03.208388 2577 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7625-140190814294352" oldPodUID="51f79b08-9a6b-4009-8bf2-d020f5b05683" podUID="6055f39d-9ddf-4561-8d4f-4beb32a2802c" Aug 04 11:21:03.533060 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:03.533033 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f79b08-9a6b-4009-8bf2-d020f5b05683" path="/var/lib/kubelet/pods/51f79b08-9a6b-4009-8bf2-d020f5b05683/volumes" Aug 04 11:21:04.204072 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:04.204041 2577 generic.go:356] "Generic (PLEG): container finished" podID="6055f39d-9ddf-4561-8d4f-4beb32a2802c" containerID="26990a113007cd3c0100ade7f2830be9e1e800fd33ddd0fe9bf9ce77c2a479da" exitCode=0 Aug 04 11:21:04.204257 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:04.204081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7625-140190814294352" event={"ID":"6055f39d-9ddf-4561-8d4f-4beb32a2802c","Type":"ContainerDied","Data":"26990a113007cd3c0100ade7f2830be9e1e800fd33ddd0fe9bf9ce77c2a479da"} Aug 04 11:21:05.346805 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.346783 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:05.369901 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.369863 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7625-140190814294352_6055f39d-9ddf-4561-8d4f-4beb32a2802c/test-curl-7625-140190814294352/0.log" Aug 04 11:21:05.401400 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.401372 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:21:05.405311 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.405289 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7625-140190814294352"] Aug 04 11:21:05.494752 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.494697 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7gq7\" (UniqueName: \"kubernetes.io/projected/6055f39d-9ddf-4561-8d4f-4beb32a2802c-kube-api-access-d7gq7\") pod \"6055f39d-9ddf-4561-8d4f-4beb32a2802c\" (UID: \"6055f39d-9ddf-4561-8d4f-4beb32a2802c\") " Aug 04 11:21:05.496732 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.496712 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6055f39d-9ddf-4561-8d4f-4beb32a2802c-kube-api-access-d7gq7" (OuterVolumeSpecName: "kube-api-access-d7gq7") pod "6055f39d-9ddf-4561-8d4f-4beb32a2802c" (UID: "6055f39d-9ddf-4561-8d4f-4beb32a2802c"). InnerVolumeSpecName "kube-api-access-d7gq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:21:05.532041 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.532019 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6055f39d-9ddf-4561-8d4f-4beb32a2802c" path="/var/lib/kubelet/pods/6055f39d-9ddf-4561-8d4f-4beb32a2802c/volumes" Aug 04 11:21:05.596443 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:05.596413 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7gq7\" (UniqueName: \"kubernetes.io/projected/6055f39d-9ddf-4561-8d4f-4beb32a2802c-kube-api-access-d7gq7\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:21:06.222128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:06.222096 2577 scope.go:117] "RemoveContainer" containerID="26990a113007cd3c0100ade7f2830be9e1e800fd33ddd0fe9bf9ce77c2a479da" Aug 04 11:21:06.222128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:06.222116 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7625-140190814294352" Aug 04 11:21:10.159863 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.159813 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx"] Aug 04 11:21:10.160472 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.160456 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6055f39d-9ddf-4561-8d4f-4beb32a2802c" containerName="test-curl-7625-140190814294352" Aug 04 11:21:10.160530 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.160476 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6055f39d-9ddf-4561-8d4f-4beb32a2802c" containerName="test-curl-7625-140190814294352" Aug 04 11:21:10.160595 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.160584 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6055f39d-9ddf-4561-8d4f-4beb32a2802c" containerName="test-curl-7625-140190814294352" Aug 04 11:21:10.165525 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.165506 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.167607 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.167587 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1cb57d-serving-cert\"" Aug 04 11:21:10.173581 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.173558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx"] Aug 04 11:21:10.237507 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.237479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67af70d8-bd29-4cd5-82bd-d23013cdd344-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.237610 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.237509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67af70d8-bd29-4cd5-82bd-d23013cdd344-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.237659 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.237632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddqp\" (UniqueName: \"kubernetes.io/projected/67af70d8-bd29-4cd5-82bd-d23013cdd344-kube-api-access-bddqp\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.338843 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.338812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddqp\" (UniqueName: \"kubernetes.io/projected/67af70d8-bd29-4cd5-82bd-d23013cdd344-kube-api-access-bddqp\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.338975 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.338915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67af70d8-bd29-4cd5-82bd-d23013cdd344-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.338975 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.338938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67af70d8-bd29-4cd5-82bd-d23013cdd344-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.339650 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.339629 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67af70d8-bd29-4cd5-82bd-d23013cdd344-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.341366 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.341346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67af70d8-bd29-4cd5-82bd-d23013cdd344-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.346197 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.346178 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddqp\" (UniqueName: \"kubernetes.io/projected/67af70d8-bd29-4cd5-82bd-d23013cdd344-kube-api-access-bddqp\") pod \"maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.477304 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.477217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:10.614996 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:10.614969 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx"] Aug 04 11:21:10.617192 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:21:10.617166 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67af70d8_bd29_4cd5_82bd_d23013cdd344.slice/crio-e600deee3399dee797d5249f65092278a1680cc4b07f890c65fdc6f3a73f0222 WatchSource:0}: Error finding container e600deee3399dee797d5249f65092278a1680cc4b07f890c65fdc6f3a73f0222: Status 404 returned error can't find the container with id e600deee3399dee797d5249f65092278a1680cc4b07f890c65fdc6f3a73f0222 Aug 04 11:21:11.254209 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:11.254180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" event={"ID":"67af70d8-bd29-4cd5-82bd-d23013cdd344","Type":"ContainerStarted","Data":"ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7"} Aug 04 11:21:11.254549 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:11.254233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" event={"ID":"67af70d8-bd29-4cd5-82bd-d23013cdd344","Type":"ContainerStarted","Data":"e600deee3399dee797d5249f65092278a1680cc4b07f890c65fdc6f3a73f0222"} Aug 04 11:21:11.254549 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:11.254260 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:11.272413 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:11.272372 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" podStartSLOduration=0.797225614 podStartE2EDuration="1.272356307s" totalImagesPullingTime="475.130693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:21:11.270122024 +0000 UTC m=+3232.221529572" watchObservedRunningTime="2026-08-04 11:21:11.272356307 +0000 UTC m=+3232.223763856" Aug 04 11:21:17.265218 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:17.265136 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:21:39.264381 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.264341 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg"] Aug 04 11:21:39.268144 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.268122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.270417 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.270397 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-651a7a-serving-cert\"" Aug 04 11:21:39.278882 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.278855 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg"] Aug 04 11:21:39.299259 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.299225 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2592f911-e544-4c73-9869-6e9698c1d1c0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.299394 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.299332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.299394 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.299360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv46k\" (UniqueName: \"kubernetes.io/projected/2592f911-e544-4c73-9869-6e9698c1d1c0-kube-api-access-tv46k\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.400537 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.400504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.400537 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.400539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv46k\" (UniqueName: \"kubernetes.io/projected/2592f911-e544-4c73-9869-6e9698c1d1c0-kube-api-access-tv46k\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.400772 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.400620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2592f911-e544-4c73-9869-6e9698c1d1c0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.400772 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:21:39.400655 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-serving-cert: secret "maas-api-e2e-ipp-b-651a7a-serving-cert" not found Aug 04 11:21:39.400772 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:21:39.400722 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls podName:2592f911-e544-4c73-9869-6e9698c1d1c0 nodeName:}" failed. No retries permitted until 2026-08-04 11:21:39.900705742 +0000 UTC m=+3260.852113270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls") pod "maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" (UID: "2592f911-e544-4c73-9869-6e9698c1d1c0") : secret "maas-api-e2e-ipp-b-651a7a-serving-cert" not found Aug 04 11:21:39.401279 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.401261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2592f911-e544-4c73-9869-6e9698c1d1c0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.408738 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.408709 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv46k\" (UniqueName: \"kubernetes.io/projected/2592f911-e544-4c73-9869-6e9698c1d1c0-kube-api-access-tv46k\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.905322 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.905289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:39.907737 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:39.907706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:40.181603 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:40.181517 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:40.354283 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:40.353809 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg"] Aug 04 11:21:40.420650 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:40.420616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" event={"ID":"2592f911-e544-4c73-9869-6e9698c1d1c0","Type":"ContainerStarted","Data":"f71a6b95e2a1a658d3ee8fe5028f8cff898610d49991cf574973047b4a107b72"} Aug 04 11:21:41.428869 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:41.428812 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" event={"ID":"2592f911-e544-4c73-9869-6e9698c1d1c0","Type":"ContainerStarted","Data":"264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3"} Aug 04 11:21:41.429482 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:41.428898 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:21:41.445004 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:41.444951 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" podStartSLOduration=1.9663209940000002 podStartE2EDuration="2.444933211s" totalImagesPullingTime="478.612217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:21:41.444125931 +0000 UTC m=+3262.395533481" watchObservedRunningTime="2026-08-04 11:21:41.444933211 +0000 UTC m=+3262.396340762" Aug 04 11:21:47.440115 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:21:47.440088 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:22:19.883844 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:19.883800 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:22:19.893214 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:19.893191 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:22:55.895814 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:55.895738 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh"] Aug 04 11:22:55.904233 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:55.904202 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:55.906577 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:55.906555 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a031b5-openshift-default-dockercfg-fnjn7\"" Aug 04 11:22:55.906713 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:55.906556 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 11:22:55.911419 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:55.911388 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh"] Aug 04 11:22:56.022652 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022618 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-data\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.022652 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da78152a-206d-40a3-8dc2-c86de333750d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.022901 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-istio-token\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.022901 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da78152a-206d-40a3-8dc2-c86de333750d-istio-podinfo\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.022901 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-certs\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.022901 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-envoy\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.023031 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85knq\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-kube-api-access-85knq\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.023031 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-socket\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.023031 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.022998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-credential-socket\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.124527 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da78152a-206d-40a3-8dc2-c86de333750d-istio-podinfo\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.124527 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-certs\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.124763 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-envoy\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.124763 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85knq\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-kube-api-access-85knq\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.124763 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-socket\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.124763 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-credential-socket\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.124763 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-data\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.125052 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da78152a-206d-40a3-8dc2-c86de333750d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.125052 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-istio-token\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.125052 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.124954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-certs\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.125411 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.125265 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-socket\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.125411 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.125336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-credential-socket\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.125411 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.125400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-data\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.125658 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.125635 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da78152a-206d-40a3-8dc2-c86de333750d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.127012 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.126994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-envoy\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.127971 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.127955 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da78152a-206d-40a3-8dc2-c86de333750d-istio-podinfo\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.133277 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.133252 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85knq\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-kube-api-access-85knq\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.133370 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.133254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-istio-token\") pod \"e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.216604 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.216518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:22:56.359926 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.359895 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh"] Aug 04 11:22:56.361547 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:22:56.361518 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda78152a_206d_40a3_8dc2_c86de333750d.slice/crio-eaa1a165ff3a865aca917514f3ad34a2a8691db1252460aced42d19d512a384d WatchSource:0}: Error finding container eaa1a165ff3a865aca917514f3ad34a2a8691db1252460aced42d19d512a384d: Status 404 returned error can't find the container with id eaa1a165ff3a865aca917514f3ad34a2a8691db1252460aced42d19d512a384d Aug 04 11:22:56.859616 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:56.859577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" event={"ID":"da78152a-206d-40a3-8dc2-c86de333750d","Type":"ContainerStarted","Data":"eaa1a165ff3a865aca917514f3ad34a2a8691db1252460aced42d19d512a384d"} Aug 04 11:22:59.253011 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:59.252973 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 11:22:59.253384 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:59.253060 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 11:22:59.253384 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:59.253108 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 11:22:59.881193 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:59.881157 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" event={"ID":"da78152a-206d-40a3-8dc2-c86de333750d","Type":"ContainerStarted","Data":"dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57"} Aug 04 11:22:59.902029 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:22:59.901973 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podStartSLOduration=2.013101668 podStartE2EDuration="4.901955038s" totalImagesPullingTime="2.88885337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:22:59.89941619 +0000 UTC m=+3340.850823739" watchObservedRunningTime="2026-08-04 11:22:59.901955038 +0000 UTC m=+3340.853362587" Aug 04 11:23:00.216983 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:00.216898 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:00.221984 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:00.221961 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:00.887871 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:00.887816 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:00.888991 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:00.888974 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:02.928905 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:02.928874 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w"] Aug 04 11:23:02.932560 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:02.932542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:02.934398 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:02.934377 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a031b5-serving-cert\"" Aug 04 11:23:02.943942 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:02.943913 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w"] Aug 04 11:23:02.988503 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:02.988468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1352d3e-37b3-4e94-868e-67270502992f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:02.988681 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:02.988592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1352d3e-37b3-4e94-868e-67270502992f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:02.988761 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:02.988736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8dm\" (UniqueName: \"kubernetes.io/projected/c1352d3e-37b3-4e94-868e-67270502992f-kube-api-access-9l8dm\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.090100 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.090067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1352d3e-37b3-4e94-868e-67270502992f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.090285 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.090124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8dm\" (UniqueName: \"kubernetes.io/projected/c1352d3e-37b3-4e94-868e-67270502992f-kube-api-access-9l8dm\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.090285 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.090200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1352d3e-37b3-4e94-868e-67270502992f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.090882 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.090861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1352d3e-37b3-4e94-868e-67270502992f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.092515 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.092498 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1352d3e-37b3-4e94-868e-67270502992f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.109524 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.109500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8dm\" (UniqueName: \"kubernetes.io/projected/c1352d3e-37b3-4e94-868e-67270502992f-kube-api-access-9l8dm\") pod \"maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.243336 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.243251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.377674 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.377647 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w"] Aug 04 11:23:03.379936 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:23:03.379904 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1352d3e_37b3_4e94_868e_67270502992f.slice/crio-4e1b24c09d76b6d0cd561315f5a8a5d71ce56abf4723e192843a5482ced09fd6 WatchSource:0}: Error finding container 4e1b24c09d76b6d0cd561315f5a8a5d71ce56abf4723e192843a5482ced09fd6: Status 404 returned error can't find the container with id 4e1b24c09d76b6d0cd561315f5a8a5d71ce56abf4723e192843a5482ced09fd6 Aug 04 11:23:03.906707 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.906672 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" event={"ID":"c1352d3e-37b3-4e94-868e-67270502992f","Type":"ContainerStarted","Data":"5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98"} Aug 04 11:23:03.906897 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.906734 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:03.906897 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.906751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" event={"ID":"c1352d3e-37b3-4e94-868e-67270502992f","Type":"ContainerStarted","Data":"4e1b24c09d76b6d0cd561315f5a8a5d71ce56abf4723e192843a5482ced09fd6"} Aug 04 11:23:03.924574 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:03.924529 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" podStartSLOduration=1.562516511 podStartE2EDuration="1.924515424s" totalImagesPullingTime="361.998913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 11:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:23:03.923107891 +0000 UTC m=+3344.874515443" watchObservedRunningTime="2026-08-04 11:23:03.924515424 +0000 UTC m=+3344.875922972" Aug 04 11:23:09.918127 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:09.918089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:15.889985 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.889949 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Aug 04 11:23:15.890390 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.890002 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Aug 04 11:23:15.982724 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.982691 2577 generic.go:356] "Generic (PLEG): container finished" podID="da78152a-206d-40a3-8dc2-c86de333750d" containerID="dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57" exitCode=0 Aug 04 11:23:15.982927 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.982742 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" event={"ID":"da78152a-206d-40a3-8dc2-c86de333750d","Type":"ContainerDied","Data":"dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57"} Aug 04 11:23:15.983160 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.983143 2577 scope.go:117] "RemoveContainer" containerID="dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57" Aug 04 11:23:15.985161 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.985134 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 11:23:15.985226 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.985198 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 11:23:15.985277 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:15.985231 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 11:23:16.992632 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:16.990342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" event={"ID":"da78152a-206d-40a3-8dc2-c86de333750d","Type":"ContainerStarted","Data":"40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3"} Aug 04 11:23:17.217389 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:17.217354 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:17.217556 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:17.217519 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Aug 04 11:23:17.217607 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:17.217571 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Aug 04 11:23:18.217794 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:18.217758 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Aug 04 11:23:18.218204 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:18.217812 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Aug 04 11:23:19.217474 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:19.217433 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Aug 04 11:23:19.217668 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:19.217492 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Aug 04 11:23:20.217014 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:20.216979 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Aug 04 11:23:20.217418 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:20.217043 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Aug 04 11:23:21.217819 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:21.217785 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Aug 04 11:23:21.218233 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:21.217902 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Aug 04 11:23:22.221246 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:22.221216 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:22.221703 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:22.221519 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:22.222199 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:22.222183 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:32.009054 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.009011 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w"] Aug 04 11:23:32.009620 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.009216 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" podUID="c1352d3e-37b3-4e94-868e-67270502992f" containerName="maas-api" containerID="cri-o://5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98" gracePeriod=30 Aug 04 11:23:32.265494 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.265433 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:32.394336 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.394294 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1352d3e-37b3-4e94-868e-67270502992f-maas-api-tls\") pod \"c1352d3e-37b3-4e94-868e-67270502992f\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " Aug 04 11:23:32.394528 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.394380 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l8dm\" (UniqueName: \"kubernetes.io/projected/c1352d3e-37b3-4e94-868e-67270502992f-kube-api-access-9l8dm\") pod \"c1352d3e-37b3-4e94-868e-67270502992f\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " Aug 04 11:23:32.394528 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.394427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1352d3e-37b3-4e94-868e-67270502992f-openshift-service-ca\") pod \"c1352d3e-37b3-4e94-868e-67270502992f\" (UID: \"c1352d3e-37b3-4e94-868e-67270502992f\") " Aug 04 11:23:32.394847 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.394805 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1352d3e-37b3-4e94-868e-67270502992f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1352d3e-37b3-4e94-868e-67270502992f" (UID: "c1352d3e-37b3-4e94-868e-67270502992f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:23:32.396425 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.396396 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1352d3e-37b3-4e94-868e-67270502992f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1352d3e-37b3-4e94-868e-67270502992f" (UID: "c1352d3e-37b3-4e94-868e-67270502992f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:23:32.396425 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.396415 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1352d3e-37b3-4e94-868e-67270502992f-kube-api-access-9l8dm" (OuterVolumeSpecName: "kube-api-access-9l8dm") pod "c1352d3e-37b3-4e94-868e-67270502992f" (UID: "c1352d3e-37b3-4e94-868e-67270502992f"). InnerVolumeSpecName "kube-api-access-9l8dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:23:32.495840 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.495794 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l8dm\" (UniqueName: \"kubernetes.io/projected/c1352d3e-37b3-4e94-868e-67270502992f-kube-api-access-9l8dm\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:32.495996 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.495825 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1352d3e-37b3-4e94-868e-67270502992f-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:32.495996 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:32.495860 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1352d3e-37b3-4e94-868e-67270502992f-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:33.085094 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.085056 2577 generic.go:356] "Generic (PLEG): container finished" podID="c1352d3e-37b3-4e94-868e-67270502992f" containerID="5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98" exitCode=0 Aug 04 11:23:33.085509 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.085108 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" Aug 04 11:23:33.085509 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.085141 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" event={"ID":"c1352d3e-37b3-4e94-868e-67270502992f","Type":"ContainerDied","Data":"5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98"} Aug 04 11:23:33.085509 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.085193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w" event={"ID":"c1352d3e-37b3-4e94-868e-67270502992f","Type":"ContainerDied","Data":"4e1b24c09d76b6d0cd561315f5a8a5d71ce56abf4723e192843a5482ced09fd6"} Aug 04 11:23:33.085509 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.085216 2577 scope.go:117] "RemoveContainer" containerID="5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98" Aug 04 11:23:33.095735 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.095712 2577 scope.go:117] "RemoveContainer" containerID="5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98" Aug 04 11:23:33.096018 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:23:33.095998 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98\": container with ID starting with 5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98 not found: ID does not exist" containerID="5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98" Aug 04 11:23:33.096087 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.096030 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98"} err="failed to get container status \"5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98\": rpc error: code = NotFound desc = could not find container \"5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98\": container with ID starting with 5ff77b29134c9c25a9665a0e590ee472150a75fe7a462489b7e533849aac8d98 not found: ID does not exist" Aug 04 11:23:33.105970 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.105948 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w"] Aug 04 11:23:33.109604 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.109581 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a031b5-d9cb855b7-zfh2w"] Aug 04 11:23:33.532915 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:33.532887 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1352d3e-37b3-4e94-868e-67270502992f" path="/var/lib/kubelet/pods/c1352d3e-37b3-4e94-868e-67270502992f/volumes" Aug 04 11:23:39.171169 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:39.171136 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh"] Aug 04 11:23:39.171609 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:39.171336 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" containerID="cri-o://40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3" gracePeriod=30 Aug 04 11:23:44.419470 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.419441 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:44.495229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495141 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-credential-socket\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495178 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85knq\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-kube-api-access-85knq\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495229 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495197 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-istio-token\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495467 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495247 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-envoy\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495467 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495303 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-socket\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495467 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495342 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da78152a-206d-40a3-8dc2-c86de333750d-istiod-ca-cert\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495467 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495371 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da78152a-206d-40a3-8dc2-c86de333750d-istio-podinfo\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495467 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-data\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495467 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495446 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-certs\") pod \"da78152a-206d-40a3-8dc2-c86de333750d\" (UID: \"da78152a-206d-40a3-8dc2-c86de333750d\") " Aug 04 11:23:44.495740 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495477 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:44.495803 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495759 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-credential-socket\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.495803 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495780 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-data" (OuterVolumeSpecName: "istio-data") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:44.495950 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495873 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:44.496005 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.495972 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:44.496062 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.496021 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da78152a-206d-40a3-8dc2-c86de333750d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:23:44.498164 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.498144 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-istio-token" (OuterVolumeSpecName: "istio-token") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:23:44.498276 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.498177 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-kube-api-access-85knq" (OuterVolumeSpecName: "kube-api-access-85knq") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "kube-api-access-85knq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:23:44.498276 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.498184 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da78152a-206d-40a3-8dc2-c86de333750d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 11:23:44.498276 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.498184 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da78152a-206d-40a3-8dc2-c86de333750d" (UID: "da78152a-206d-40a3-8dc2-c86de333750d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 11:23:44.597209 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597168 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-data\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.597209 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597200 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-certs\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.597209 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597210 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85knq\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-kube-api-access-85knq\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.597209 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597223 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da78152a-206d-40a3-8dc2-c86de333750d-istio-token\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.597535 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597232 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-istio-envoy\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.597535 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597241 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da78152a-206d-40a3-8dc2-c86de333750d-workload-socket\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.597535 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597253 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da78152a-206d-40a3-8dc2-c86de333750d-istiod-ca-cert\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:44.597535 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:44.597265 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da78152a-206d-40a3-8dc2-c86de333750d-istio-podinfo\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:45.159956 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.159924 2577 generic.go:356] "Generic (PLEG): container finished" podID="da78152a-206d-40a3-8dc2-c86de333750d" containerID="40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3" exitCode=0 Aug 04 11:23:45.160133 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.159972 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" Aug 04 11:23:45.160133 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.160009 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" event={"ID":"da78152a-206d-40a3-8dc2-c86de333750d","Type":"ContainerDied","Data":"40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3"} Aug 04 11:23:45.160133 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.160061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh" event={"ID":"da78152a-206d-40a3-8dc2-c86de333750d","Type":"ContainerDied","Data":"eaa1a165ff3a865aca917514f3ad34a2a8691db1252460aced42d19d512a384d"} Aug 04 11:23:45.160133 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.160082 2577 scope.go:117] "RemoveContainer" containerID="40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3" Aug 04 11:23:45.169691 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.169675 2577 scope.go:117] "RemoveContainer" containerID="dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57" Aug 04 11:23:45.177861 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.177843 2577 scope.go:117] "RemoveContainer" containerID="40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3" Aug 04 11:23:45.178105 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:23:45.178086 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3\": container with ID starting with 40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3 not found: ID does not exist" containerID="40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3" Aug 04 11:23:45.178178 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.178113 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3"} err="failed to get container status \"40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3\": rpc error: code = NotFound desc = could not find container \"40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3\": container with ID starting with 40b5009d46232f302eb1ffc4281af347e6a6a986b2fbdc2b9505cd7d2e6d4fe3 not found: ID does not exist" Aug 04 11:23:45.178178 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.178134 2577 scope.go:117] "RemoveContainer" containerID="dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57" Aug 04 11:23:45.178409 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:23:45.178378 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57\": container with ID starting with dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57 not found: ID does not exist" containerID="dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57" Aug 04 11:23:45.178486 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.178418 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57"} err="failed to get container status \"dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57\": rpc error: code = NotFound desc = could not find container \"dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57\": container with ID starting with dfd1fc7baf4c5a18bfb4a2c5774005ba961f090fcf031a9dd06eac16e5515b57 not found: ID does not exist" Aug 04 11:23:45.181466 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.181447 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh"] Aug 04 11:23:45.185624 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.185602 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a031b5-openshift-default-785f9965bd-hcjdh"] Aug 04 11:23:45.533217 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:45.533183 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da78152a-206d-40a3-8dc2-c86de333750d" path="/var/lib/kubelet/pods/da78152a-206d-40a3-8dc2-c86de333750d/volumes" Aug 04 11:23:58.405126 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.405085 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx"] Aug 04 11:23:58.405542 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.405282 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" podUID="67af70d8-bd29-4cd5-82bd-d23013cdd344" containerName="maas-api" containerID="cri-o://ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7" gracePeriod=30 Aug 04 11:23:58.666249 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.666189 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:23:58.728454 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.728426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bddqp\" (UniqueName: \"kubernetes.io/projected/67af70d8-bd29-4cd5-82bd-d23013cdd344-kube-api-access-bddqp\") pod \"67af70d8-bd29-4cd5-82bd-d23013cdd344\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " Aug 04 11:23:58.728615 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.728492 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67af70d8-bd29-4cd5-82bd-d23013cdd344-openshift-service-ca\") pod \"67af70d8-bd29-4cd5-82bd-d23013cdd344\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " Aug 04 11:23:58.728615 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.728518 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67af70d8-bd29-4cd5-82bd-d23013cdd344-maas-api-tls\") pod \"67af70d8-bd29-4cd5-82bd-d23013cdd344\" (UID: \"67af70d8-bd29-4cd5-82bd-d23013cdd344\") " Aug 04 11:23:58.728906 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.728881 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67af70d8-bd29-4cd5-82bd-d23013cdd344-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67af70d8-bd29-4cd5-82bd-d23013cdd344" (UID: "67af70d8-bd29-4cd5-82bd-d23013cdd344"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:23:58.730495 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.730472 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67af70d8-bd29-4cd5-82bd-d23013cdd344-kube-api-access-bddqp" (OuterVolumeSpecName: "kube-api-access-bddqp") pod "67af70d8-bd29-4cd5-82bd-d23013cdd344" (UID: "67af70d8-bd29-4cd5-82bd-d23013cdd344"). InnerVolumeSpecName "kube-api-access-bddqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:23:58.730582 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.730514 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67af70d8-bd29-4cd5-82bd-d23013cdd344-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67af70d8-bd29-4cd5-82bd-d23013cdd344" (UID: "67af70d8-bd29-4cd5-82bd-d23013cdd344"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:23:58.830250 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.830220 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bddqp\" (UniqueName: \"kubernetes.io/projected/67af70d8-bd29-4cd5-82bd-d23013cdd344-kube-api-access-bddqp\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:58.830250 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.830245 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67af70d8-bd29-4cd5-82bd-d23013cdd344-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:58.830250 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:58.830255 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67af70d8-bd29-4cd5-82bd-d23013cdd344-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:23:59.247706 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.247668 2577 generic.go:356] "Generic (PLEG): container finished" podID="67af70d8-bd29-4cd5-82bd-d23013cdd344" containerID="ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7" exitCode=0 Aug 04 11:23:59.247907 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.247730 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" Aug 04 11:23:59.247907 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.247757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" event={"ID":"67af70d8-bd29-4cd5-82bd-d23013cdd344","Type":"ContainerDied","Data":"ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7"} Aug 04 11:23:59.247907 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.247864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx" event={"ID":"67af70d8-bd29-4cd5-82bd-d23013cdd344","Type":"ContainerDied","Data":"e600deee3399dee797d5249f65092278a1680cc4b07f890c65fdc6f3a73f0222"} Aug 04 11:23:59.247907 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.247895 2577 scope.go:117] "RemoveContainer" containerID="ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7" Aug 04 11:23:59.257673 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.257656 2577 scope.go:117] "RemoveContainer" containerID="ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7" Aug 04 11:23:59.257921 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:23:59.257901 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7\": container with ID starting with ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7 not found: ID does not exist" containerID="ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7" Aug 04 11:23:59.258001 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.257933 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7"} err="failed to get container status \"ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7\": rpc error: code = NotFound desc = could not find container \"ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7\": container with ID starting with ddf6ab4d128667eaa1c630c2353e1489ee8248748c0eefcd287778b0dd783ca7 not found: ID does not exist" Aug 04 11:23:59.269410 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.269382 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx"] Aug 04 11:23:59.272826 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.272806 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1cb57d-8d7b7c88-x7qwx"] Aug 04 11:23:59.533982 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:23:59.533950 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67af70d8-bd29-4cd5-82bd-d23013cdd344" path="/var/lib/kubelet/pods/67af70d8-bd29-4cd5-82bd-d23013cdd344/volumes" Aug 04 11:24:59.138361 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.138327 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg"] Aug 04 11:24:59.138983 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.138590 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" podUID="2592f911-e544-4c73-9869-6e9698c1d1c0" containerName="maas-api" containerID="cri-o://264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3" gracePeriod=30 Aug 04 11:24:59.392322 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.392252 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:24:59.463513 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.463477 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv46k\" (UniqueName: \"kubernetes.io/projected/2592f911-e544-4c73-9869-6e9698c1d1c0-kube-api-access-tv46k\") pod \"2592f911-e544-4c73-9869-6e9698c1d1c0\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " Aug 04 11:24:59.463689 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.463538 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2592f911-e544-4c73-9869-6e9698c1d1c0-openshift-service-ca\") pod \"2592f911-e544-4c73-9869-6e9698c1d1c0\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " Aug 04 11:24:59.463689 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.463623 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls\") pod \"2592f911-e544-4c73-9869-6e9698c1d1c0\" (UID: \"2592f911-e544-4c73-9869-6e9698c1d1c0\") " Aug 04 11:24:59.463986 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.463960 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2592f911-e544-4c73-9869-6e9698c1d1c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2592f911-e544-4c73-9869-6e9698c1d1c0" (UID: "2592f911-e544-4c73-9869-6e9698c1d1c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:24:59.465614 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.465592 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2592f911-e544-4c73-9869-6e9698c1d1c0-kube-api-access-tv46k" (OuterVolumeSpecName: "kube-api-access-tv46k") pod "2592f911-e544-4c73-9869-6e9698c1d1c0" (UID: "2592f911-e544-4c73-9869-6e9698c1d1c0"). InnerVolumeSpecName "kube-api-access-tv46k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:24:59.465682 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.465623 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2592f911-e544-4c73-9869-6e9698c1d1c0" (UID: "2592f911-e544-4c73-9869-6e9698c1d1c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:24:59.565322 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.565286 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv46k\" (UniqueName: \"kubernetes.io/projected/2592f911-e544-4c73-9869-6e9698c1d1c0-kube-api-access-tv46k\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:24:59.565322 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.565314 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2592f911-e544-4c73-9869-6e9698c1d1c0-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:24:59.565322 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.565325 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2592f911-e544-4c73-9869-6e9698c1d1c0-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:24:59.591917 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.591882 2577 generic.go:356] "Generic (PLEG): container finished" podID="2592f911-e544-4c73-9869-6e9698c1d1c0" containerID="264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3" exitCode=0 Aug 04 11:24:59.592095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.591933 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" Aug 04 11:24:59.592095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.591970 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" event={"ID":"2592f911-e544-4c73-9869-6e9698c1d1c0","Type":"ContainerDied","Data":"264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3"} Aug 04 11:24:59.592095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.592023 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg" event={"ID":"2592f911-e544-4c73-9869-6e9698c1d1c0","Type":"ContainerDied","Data":"f71a6b95e2a1a658d3ee8fe5028f8cff898610d49991cf574973047b4a107b72"} Aug 04 11:24:59.592095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.592044 2577 scope.go:117] "RemoveContainer" containerID="264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3" Aug 04 11:24:59.601230 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.601210 2577 scope.go:117] "RemoveContainer" containerID="264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3" Aug 04 11:24:59.601490 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:24:59.601461 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3\": container with ID starting with 264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3 not found: ID does not exist" containerID="264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3" Aug 04 11:24:59.601592 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.601502 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3"} err="failed to get container status \"264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3\": rpc error: code = NotFound desc = could not find container \"264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3\": container with ID starting with 264d0dcef1b7107d910a3d65433fa93b515ab2fac885056f2d0563b579d12cc3 not found: ID does not exist" Aug 04 11:24:59.609058 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.609033 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg"] Aug 04 11:24:59.611212 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:24:59.611192 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-651a7a-b844b474-mmwzg"] Aug 04 11:25:01.532865 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:01.532813 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2592f911-e544-4c73-9869-6e9698c1d1c0" path="/var/lib/kubelet/pods/2592f911-e544-4c73-9869-6e9698c1d1c0/volumes" Aug 04 11:25:17.171170 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.171135 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg"] Aug 04 11:25:17.171599 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.171341 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" podUID="f9a4d994-3e68-4e60-8a3f-e5db386b17f8" containerName="maas-api" containerID="cri-o://4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3" gracePeriod=30 Aug 04 11:25:17.425567 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.425509 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:25:17.527461 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.527422 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-maas-api-tls\") pod \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " Aug 04 11:25:17.527642 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.527478 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-openshift-service-ca\") pod \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " Aug 04 11:25:17.527642 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.527580 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7tps\" (UniqueName: \"kubernetes.io/projected/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-kube-api-access-z7tps\") pod \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\" (UID: \"f9a4d994-3e68-4e60-8a3f-e5db386b17f8\") " Aug 04 11:25:17.527988 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.527936 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9a4d994-3e68-4e60-8a3f-e5db386b17f8" (UID: "f9a4d994-3e68-4e60-8a3f-e5db386b17f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:25:17.530665 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.530638 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9a4d994-3e68-4e60-8a3f-e5db386b17f8" (UID: "f9a4d994-3e68-4e60-8a3f-e5db386b17f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:25:17.530856 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.530666 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-kube-api-access-z7tps" (OuterVolumeSpecName: "kube-api-access-z7tps") pod "f9a4d994-3e68-4e60-8a3f-e5db386b17f8" (UID: "f9a4d994-3e68-4e60-8a3f-e5db386b17f8"). InnerVolumeSpecName "kube-api-access-z7tps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:17.629271 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.629237 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:25:17.629271 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.629266 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:25:17.629271 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.629275 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7tps\" (UniqueName: \"kubernetes.io/projected/f9a4d994-3e68-4e60-8a3f-e5db386b17f8-kube-api-access-z7tps\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:25:17.691084 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.690994 2577 generic.go:356] "Generic (PLEG): container finished" podID="f9a4d994-3e68-4e60-8a3f-e5db386b17f8" containerID="4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3" exitCode=0 Aug 04 11:25:17.691084 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.691041 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" Aug 04 11:25:17.691330 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.691081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" event={"ID":"f9a4d994-3e68-4e60-8a3f-e5db386b17f8","Type":"ContainerDied","Data":"4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3"} Aug 04 11:25:17.691330 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.691140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg" event={"ID":"f9a4d994-3e68-4e60-8a3f-e5db386b17f8","Type":"ContainerDied","Data":"22109639c8d577febc2b5bd24aa83af34ebbd1af871314d21804122d0dd21acf"} Aug 04 11:25:17.691330 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.691167 2577 scope.go:117] "RemoveContainer" containerID="4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3" Aug 04 11:25:17.700089 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.700067 2577 scope.go:117] "RemoveContainer" containerID="4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3" Aug 04 11:25:17.700349 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:25:17.700329 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3\": container with ID starting with 4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3 not found: ID does not exist" containerID="4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3" Aug 04 11:25:17.700414 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.700360 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3"} err="failed to get container status \"4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3\": rpc error: code = NotFound desc = could not find container \"4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3\": container with ID starting with 4e7c640fcf07f46b20f2a73fb27ee9167682966ed5c8f32a0e855d3716c3bea3 not found: ID does not exist" Aug 04 11:25:17.707413 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.707387 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg"] Aug 04 11:25:17.711591 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:17.711568 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bc4a78-6f48f7d88c-p5mpg"] Aug 04 11:25:19.533625 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:19.533591 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a4d994-3e68-4e60-8a3f-e5db386b17f8" path="/var/lib/kubelet/pods/f9a4d994-3e68-4e60-8a3f-e5db386b17f8/volumes" Aug 04 11:25:34.967125 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:34.967083 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt"] Aug 04 11:25:34.967685 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:34.967333 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" podUID="4767125a-878e-4490-b3d2-69eb14ca063d" containerName="maas-api" containerID="cri-o://ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037" gracePeriod=30 Aug 04 11:25:35.214975 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.214953 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:25:35.297180 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.297153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls\") pod \"4767125a-878e-4490-b3d2-69eb14ca063d\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " Aug 04 11:25:35.297315 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.297190 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4767125a-878e-4490-b3d2-69eb14ca063d-openshift-service-ca\") pod \"4767125a-878e-4490-b3d2-69eb14ca063d\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " Aug 04 11:25:35.297315 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.297237 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrwbx\" (UniqueName: \"kubernetes.io/projected/4767125a-878e-4490-b3d2-69eb14ca063d-kube-api-access-qrwbx\") pod \"4767125a-878e-4490-b3d2-69eb14ca063d\" (UID: \"4767125a-878e-4490-b3d2-69eb14ca063d\") " Aug 04 11:25:35.297573 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.297546 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4767125a-878e-4490-b3d2-69eb14ca063d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4767125a-878e-4490-b3d2-69eb14ca063d" (UID: "4767125a-878e-4490-b3d2-69eb14ca063d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 11:25:35.299264 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.299245 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4767125a-878e-4490-b3d2-69eb14ca063d" (UID: "4767125a-878e-4490-b3d2-69eb14ca063d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 11:25:35.299405 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.299385 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4767125a-878e-4490-b3d2-69eb14ca063d-kube-api-access-qrwbx" (OuterVolumeSpecName: "kube-api-access-qrwbx") pod "4767125a-878e-4490-b3d2-69eb14ca063d" (UID: "4767125a-878e-4490-b3d2-69eb14ca063d"). InnerVolumeSpecName "kube-api-access-qrwbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 11:25:35.398490 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.398469 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4767125a-878e-4490-b3d2-69eb14ca063d-maas-api-tls\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:25:35.398490 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.398490 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4767125a-878e-4490-b3d2-69eb14ca063d-openshift-service-ca\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:25:35.398644 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.398501 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrwbx\" (UniqueName: \"kubernetes.io/projected/4767125a-878e-4490-b3d2-69eb14ca063d-kube-api-access-qrwbx\") on node \"ip-10-0-139-198.ec2.internal\" DevicePath \"\"" Aug 04 11:25:35.790023 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.789992 2577 generic.go:356] "Generic (PLEG): container finished" podID="4767125a-878e-4490-b3d2-69eb14ca063d" containerID="ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037" exitCode=0 Aug 04 11:25:35.790203 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.790044 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" Aug 04 11:25:35.790203 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.790070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" event={"ID":"4767125a-878e-4490-b3d2-69eb14ca063d","Type":"ContainerDied","Data":"ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037"} Aug 04 11:25:35.790203 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.790120 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt" event={"ID":"4767125a-878e-4490-b3d2-69eb14ca063d","Type":"ContainerDied","Data":"d97885215b1ecb3fbe2af723e53eea4691d50c565b9c317f0751e34e71bc5dd6"} Aug 04 11:25:35.790203 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.790142 2577 scope.go:117] "RemoveContainer" containerID="ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037" Aug 04 11:25:35.798876 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.798854 2577 scope.go:117] "RemoveContainer" containerID="ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037" Aug 04 11:25:35.799121 ip-10-0-139-198 kubenswrapper[2577]: E0804 11:25:35.799099 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037\": container with ID starting with ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037 not found: ID does not exist" containerID="ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037" Aug 04 11:25:35.799185 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.799132 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037"} err="failed to get container status \"ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037\": rpc error: code = NotFound desc = could not find container \"ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037\": container with ID starting with ab7a88a5dd5df029f9900e01ef7d0e78f947ecc83a851a628732389721c3a037 not found: ID does not exist" Aug 04 11:25:35.805787 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.805763 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt"] Aug 04 11:25:35.809272 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:35.809254 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-460cac-5759cb594d-8txzt"] Aug 04 11:25:37.532690 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:37.532654 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4767125a-878e-4490-b3d2-69eb14ca063d" path="/var/lib/kubelet/pods/4767125a-878e-4490-b3d2-69eb14ca063d/volumes" Aug 04 11:25:42.731802 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:42.731712 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64798b4f48-cvjqv_5cf45ed7-efd2-499b-896d-b237805ae0cd/authorino/0.log" Aug 04 11:25:48.991944 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:48.991913 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-8lxk2_dc039fc3-f06b-4396-a1d6-7e7a5129a432/copy-manifests/0.log" Aug 04 11:25:48.998889 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:48.998865 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-8lxk2_dc039fc3-f06b-4396-a1d6-7e7a5129a432/manager/0.log" Aug 04 11:25:49.498415 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:49.498384 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7bc9f979dd-hlj6q_fe182b94-9c7d-4a3c-bde7-8fcae9ff95af/maas-api/0.log" Aug 04 11:25:50.744289 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:50.744253 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64798b4f48-cvjqv_5cf45ed7-efd2-499b-896d-b237805ae0cd/authorino/0.log" Aug 04 11:25:51.086281 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:51.086254 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-4nv98_27aecda2-b2c7-4e72-9caa-3c80cafa8bac/kuadrant-console-plugin/0.log" Aug 04 11:25:52.066324 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:52.066289 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-5zztp_006ad155-43cc-4072-9e9a-b87fb1387ae8/kube-auth-proxy/0.log" Aug 04 11:25:52.638680 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:52.638653 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-587b49775d-8b8rl_23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a/router/0.log" Aug 04 11:25:53.217706 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:53.217680 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5ff7s_afd6430d-bd8d-4b7b-8dbc-286e3a173095/main/0.log" Aug 04 11:25:53.562879 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:25:53.562849 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-vs77p_c2c1aa1a-21d9-465d-8225-1ee260a7d299/main/0.log" Aug 04 11:26:02.241418 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:02.241388 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wdm6f_12828ebf-4f28-42bb-9882-cc4f93bee964/global-pull-secret-syncer/0.log" Aug 04 11:26:02.369652 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:02.369609 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6h49x_763e93ec-a5c0-419a-8a46-05ee3fe71399/konnectivity-agent/0.log" Aug 04 11:26:02.463304 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:02.463270 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-198.ec2.internal_5230571842e399889e9775bf9720e82e/haproxy/0.log" Aug 04 11:26:07.914668 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:07.914632 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64798b4f48-cvjqv_5cf45ed7-efd2-499b-896d-b237805ae0cd/authorino/0.log" Aug 04 11:26:08.013934 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:08.013906 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-4nv98_27aecda2-b2c7-4e72-9caa-3c80cafa8bac/kuadrant-console-plugin/0.log" Aug 04 11:26:09.799537 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:09.799503 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f82a49fc-fcb3-4c3f-ba96-f96b403b40a4/alertmanager/0.log" Aug 04 11:26:09.837947 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:09.837921 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f82a49fc-fcb3-4c3f-ba96-f96b403b40a4/config-reloader/0.log" Aug 04 11:26:09.858960 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:09.858915 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f82a49fc-fcb3-4c3f-ba96-f96b403b40a4/kube-rbac-proxy-web/0.log" Aug 04 11:26:09.886071 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:09.886014 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f82a49fc-fcb3-4c3f-ba96-f96b403b40a4/kube-rbac-proxy/0.log" Aug 04 11:26:09.919551 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:09.919526 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f82a49fc-fcb3-4c3f-ba96-f96b403b40a4/kube-rbac-proxy-metric/0.log" Aug 04 11:26:09.948671 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:09.948651 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f82a49fc-fcb3-4c3f-ba96-f96b403b40a4/prom-label-proxy/0.log" Aug 04 11:26:09.977670 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:09.977649 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f82a49fc-fcb3-4c3f-ba96-f96b403b40a4/init-config-reloader/0.log" Aug 04 11:26:10.035399 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.035354 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-9ht9p_a9bc0030-6e9b-413a-a166-c4d1e9cfd458/cluster-monitoring-operator/0.log" Aug 04 11:26:10.187477 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.187384 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-h2hf5_f225b1e8-2972-4b59-b10a-4d684718e516/monitoring-plugin/0.log" Aug 04 11:26:10.309036 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.309011 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nm65z_209f6e41-5a93-4e31-ac31-03a3d3e0d068/node-exporter/0.log" Aug 04 11:26:10.332687 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.332664 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nm65z_209f6e41-5a93-4e31-ac31-03a3d3e0d068/kube-rbac-proxy/0.log" Aug 04 11:26:10.358204 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.358187 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nm65z_209f6e41-5a93-4e31-ac31-03a3d3e0d068/init-textfile/0.log" Aug 04 11:26:10.475336 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.475268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wqvtp_885a2f93-5a78-448f-ad26-a79a716fc56c/kube-rbac-proxy-main/0.log" Aug 04 11:26:10.498680 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.498656 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wqvtp_885a2f93-5a78-448f-ad26-a79a716fc56c/kube-rbac-proxy-self/0.log" Aug 04 11:26:10.518826 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.518784 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wqvtp_885a2f93-5a78-448f-ad26-a79a716fc56c/openshift-state-metrics/0.log" Aug 04 11:26:10.826130 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.826104 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc7bb5567-pr5k9_732c78c5-42d8-4d31-8d08-e7bad8a541e6/telemeter-client/0.log" Aug 04 11:26:10.850746 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.850725 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc7bb5567-pr5k9_732c78c5-42d8-4d31-8d08-e7bad8a541e6/reload/0.log" Aug 04 11:26:10.876602 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.876566 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc7bb5567-pr5k9_732c78c5-42d8-4d31-8d08-e7bad8a541e6/kube-rbac-proxy/0.log" Aug 04 11:26:10.907380 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.907355 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd5675968-grpgj_bb5ce1e9-daf7-488c-aca8-23c3a12bde78/thanos-query/0.log" Aug 04 11:26:10.925135 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.925109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd5675968-grpgj_bb5ce1e9-daf7-488c-aca8-23c3a12bde78/kube-rbac-proxy-web/0.log" Aug 04 11:26:10.943549 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.943528 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd5675968-grpgj_bb5ce1e9-daf7-488c-aca8-23c3a12bde78/kube-rbac-proxy/0.log" Aug 04 11:26:10.963679 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.963660 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd5675968-grpgj_bb5ce1e9-daf7-488c-aca8-23c3a12bde78/prom-label-proxy/0.log" Aug 04 11:26:10.983716 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:10.983685 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd5675968-grpgj_bb5ce1e9-daf7-488c-aca8-23c3a12bde78/kube-rbac-proxy-rules/0.log" Aug 04 11:26:11.001272 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:11.001256 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bd5675968-grpgj_bb5ce1e9-daf7-488c-aca8-23c3a12bde78/kube-rbac-proxy-metrics/0.log" Aug 04 11:26:12.169243 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.169211 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-l7q4x_fb209365-f444-4ec0-99e6-afef201c1ff0/networking-console-plugin/0.log" Aug 04 11:26:12.688095 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688056 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs"] Aug 04 11:26:12.688632 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688614 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4767125a-878e-4490-b3d2-69eb14ca063d" containerName="maas-api" Aug 04 11:26:12.688632 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688632 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4767125a-878e-4490-b3d2-69eb14ca063d" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688644 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a4d994-3e68-4e60-8a3f-e5db386b17f8" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688650 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a4d994-3e68-4e60-8a3f-e5db386b17f8" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688662 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2592f911-e544-4c73-9869-6e9698c1d1c0" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688668 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2592f911-e544-4c73-9869-6e9698c1d1c0" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688678 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1352d3e-37b3-4e94-868e-67270502992f" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688684 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1352d3e-37b3-4e94-868e-67270502992f" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688691 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67af70d8-bd29-4cd5-82bd-d23013cdd344" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688696 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67af70d8-bd29-4cd5-82bd-d23013cdd344" containerName="maas-api" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688702 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688707 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688717 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" Aug 04 11:26:12.688759 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688722 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" Aug 04 11:26:12.689128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688782 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a4d994-3e68-4e60-8a3f-e5db386b17f8" containerName="maas-api" Aug 04 11:26:12.689128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688790 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2592f911-e544-4c73-9869-6e9698c1d1c0" containerName="maas-api" Aug 04 11:26:12.689128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688800 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="67af70d8-bd29-4cd5-82bd-d23013cdd344" containerName="maas-api" Aug 04 11:26:12.689128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688806 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4767125a-878e-4490-b3d2-69eb14ca063d" containerName="maas-api" Aug 04 11:26:12.689128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688810 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1352d3e-37b3-4e94-868e-67270502992f" containerName="maas-api" Aug 04 11:26:12.689128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688817 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" Aug 04 11:26:12.689128 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.688824 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da78152a-206d-40a3-8dc2-c86de333750d" containerName="istio-proxy" Aug 04 11:26:12.692145 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.692131 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.694139 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.694120 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qfz6h\"/\"default-dockercfg-67krk\"" Aug 04 11:26:12.694260 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.694120 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qfz6h\"/\"openshift-service-ca.crt\"" Aug 04 11:26:12.694521 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.694501 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qfz6h\"/\"kube-root-ca.crt\"" Aug 04 11:26:12.699110 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.699084 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs"] Aug 04 11:26:12.821941 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.821915 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/2.log" Aug 04 11:26:12.828117 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.828093 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-5kbmf_b3cbfbcb-c5dd-40dc-9b57-fad40c02bd71/console-operator/3.log" Aug 04 11:26:12.833241 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.833219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-proc\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.833340 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.833261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fb2z\" (UniqueName: \"kubernetes.io/projected/00713a3a-25ab-4122-9317-51aa27fbc647-kube-api-access-2fb2z\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.833379 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.833349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-podres\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.833414 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.833392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-lib-modules\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.833452 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.833414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-sys\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934071 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-podres\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934214 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-lib-modules\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934250 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-podres\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934285 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-lib-modules\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934285 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-sys\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934352 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-sys\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934352 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-proc\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934415 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fb2z\" (UniqueName: \"kubernetes.io/projected/00713a3a-25ab-4122-9317-51aa27fbc647-kube-api-access-2fb2z\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.934448 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.934438 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00713a3a-25ab-4122-9317-51aa27fbc647-proc\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:12.943063 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:12.943003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fb2z\" (UniqueName: \"kubernetes.io/projected/00713a3a-25ab-4122-9317-51aa27fbc647-kube-api-access-2fb2z\") pod \"perf-node-gather-daemonset-2zgrs\" (UID: \"00713a3a-25ab-4122-9317-51aa27fbc647\") " pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:13.002695 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:13.002670 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:13.134880 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:13.134853 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs"] Aug 04 11:26:13.136912 ip-10-0-139-198 kubenswrapper[2577]: W0804 11:26:13.136888 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod00713a3a_25ab_4122_9317_51aa27fbc647.slice/crio-9a3da534b34a8b1713b58455420591700712de6e0b077674d26f72b8bdcc74e3 WatchSource:0}: Error finding container 9a3da534b34a8b1713b58455420591700712de6e0b077674d26f72b8bdcc74e3: Status 404 returned error can't find the container with id 9a3da534b34a8b1713b58455420591700712de6e0b077674d26f72b8bdcc74e3 Aug 04 11:26:14.004639 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:14.004599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" event={"ID":"00713a3a-25ab-4122-9317-51aa27fbc647","Type":"ContainerStarted","Data":"839ad51adb0c3b71d8d2947ae0cdc1663c154ac8de8c0f324218bfc36304175d"} Aug 04 11:26:14.005074 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:14.004683 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" event={"ID":"00713a3a-25ab-4122-9317-51aa27fbc647","Type":"ContainerStarted","Data":"9a3da534b34a8b1713b58455420591700712de6e0b077674d26f72b8bdcc74e3"} Aug 04 11:26:14.005074 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:14.004704 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:14.020740 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:14.020674 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" podStartSLOduration=2.020658622 podStartE2EDuration="2.020658622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 11:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 11:26:14.020089678 +0000 UTC m=+3534.971497227" watchObservedRunningTime="2026-08-04 11:26:14.020658622 +0000 UTC m=+3534.972066171" Aug 04 11:26:14.076251 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:14.076219 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-2wdxq_b16fa908-9c26-438c-bf68-9e44b999a1fb/volume-data-source-validator/0.log" Aug 04 11:26:15.142222 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:15.142179 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h8frs_39c508ad-842a-41f1-9fc0-79df29976a48/dns/0.log" Aug 04 11:26:15.161824 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:15.161797 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h8frs_39c508ad-842a-41f1-9fc0-79df29976a48/kube-rbac-proxy/0.log" Aug 04 11:26:15.254485 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:15.254462 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4mkf_40a3ad3e-821e-4f8f-bf56-aeedbf62b949/dns-node-resolver/0.log" Aug 04 11:26:15.843595 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:15.843551 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bznw4_a7dd768f-e27c-413b-8aeb-fd7c20330835/node-ca/0.log" Aug 04 11:26:16.967009 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:16.966979 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-5zztp_006ad155-43cc-4072-9e9a-b87fb1387ae8/kube-auth-proxy/0.log" Aug 04 11:26:17.112655 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:17.112630 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-587b49775d-8b8rl_23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a/router/0.log" Aug 04 11:26:17.930904 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:17.930869 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rfcm6_16a59ee9-ea69-403e-8ee7-06492cb61c78/serve-healthcheck-canary/0.log" Aug 04 11:26:18.417757 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:18.417730 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-ws6m5_e94db208-1619-40c0-9b02-9bfb74cb474f/insights-operator/0.log" Aug 04 11:26:18.419558 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:18.419536 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-ws6m5_e94db208-1619-40c0-9b02-9bfb74cb474f/insights-operator/1.log" Aug 04 11:26:18.440611 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:18.440584 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cfg9_be213403-b9d8-4d9d-b287-391f07ca66a0/kube-rbac-proxy/0.log" Aug 04 11:26:18.460083 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:18.460060 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cfg9_be213403-b9d8-4d9d-b287-391f07ca66a0/exporter/0.log" Aug 04 11:26:18.479192 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:18.479174 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cfg9_be213403-b9d8-4d9d-b287-391f07ca66a0/extractor/0.log" Aug 04 11:26:20.020133 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:20.020098 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qfz6h/perf-node-gather-daemonset-2zgrs" Aug 04 11:26:21.438003 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:21.437973 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-8lxk2_dc039fc3-f06b-4396-a1d6-7e7a5129a432/manager/0.log" Aug 04 11:26:21.455408 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:21.455381 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-598d67d848-8lxk2_dc039fc3-f06b-4396-a1d6-7e7a5129a432/copy-manifests/0.log" Aug 04 11:26:23.009192 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:23.009155 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59766cd755-prqg4_5edc4f13-ebeb-4ff6-80bd-b791279204a3/manager/0.log" Aug 04 11:26:23.453467 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:23.453439 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-5zztp_006ad155-43cc-4072-9e9a-b87fb1387ae8/kube-auth-proxy/0.log" Aug 04 11:26:23.547764 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:23.547737 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-587b49775d-8b8rl_23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a/router/0.log" Aug 04 11:26:25.105419 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:25.105389 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bd95bfc5-5zztp_006ad155-43cc-4072-9e9a-b87fb1387ae8/kube-auth-proxy/0.log" Aug 04 11:26:25.214986 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:25.214957 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-587b49775d-8b8rl_23dc1b44-3c3f-4aa3-a9ad-0a95f697b70a/router/0.log" Aug 04 11:26:28.405280 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:28.405249 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-4grhn_94b88d24-64e8-4d0e-98b4-96c9f17a2d85/kube-storage-version-migrator-operator/1.log" Aug 04 11:26:28.406968 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:28.406940 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-4grhn_94b88d24-64e8-4d0e-98b4-96c9f17a2d85/kube-storage-version-migrator-operator/0.log" Aug 04 11:26:29.574659 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.574627 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8jftd_4bce4a74-6633-4b34-8d8d-86c76d30dcce/kube-multus/0.log" Aug 04 11:26:29.598694 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.598660 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p82pr_0f593643-7d48-4bfd-b9c4-1ba164f23119/kube-multus-additional-cni-plugins/0.log" Aug 04 11:26:29.615319 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.615293 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p82pr_0f593643-7d48-4bfd-b9c4-1ba164f23119/egress-router-binary-copy/0.log" Aug 04 11:26:29.631477 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.631451 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p82pr_0f593643-7d48-4bfd-b9c4-1ba164f23119/cni-plugins/0.log" Aug 04 11:26:29.648077 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.648052 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p82pr_0f593643-7d48-4bfd-b9c4-1ba164f23119/bond-cni-plugin/0.log" Aug 04 11:26:29.664088 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.664069 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p82pr_0f593643-7d48-4bfd-b9c4-1ba164f23119/routeoverride-cni/0.log" Aug 04 11:26:29.683204 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.683180 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p82pr_0f593643-7d48-4bfd-b9c4-1ba164f23119/whereabouts-cni-bincopy/0.log" Aug 04 11:26:29.710552 ip-10-0-139-198 kubenswrapper[2577]: I0804 11:26:29.710528 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p82pr_0f593643-7d48-4bfd-b9c4-1ba164f23119/whereabouts-cni/0.log"