Jul 20 22:51:32.966465 ip-10-0-134-37 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 20 22:51:32.966477 ip-10-0-134-37 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 20 22:51:32.966485 ip-10-0-134-37 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 20 22:51:32.966725 ip-10-0-134-37 systemd[1]: Failed to start Kubernetes Kubelet. Jul 20 22:51:43.027091 ip-10-0-134-37 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 20 22:51:43.027104 ip-10-0-134-37 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d8db9e9b98664794bddb38e32800e2ca -- Jul 20 22:53:30.399779 ip-10-0-134-37 systemd[1]: Starting Kubernetes Kubelet... Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.730425 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736330 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736343 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736347 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736351 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736354 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736357 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736360 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:30.791800 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736362 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736365 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736368 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736370 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736373 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736384 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736387 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736390 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736393 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736395 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736398 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736400 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736403 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736405 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736408 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736411 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736413 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736416 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736419 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736422 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:30.792864 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736424 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736427 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736429 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736432 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736434 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736437 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736439 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736442 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736444 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736447 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736451 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736456 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736459 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736462 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736464 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736468 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736470 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736473 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736475 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:30.793755 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736478 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736480 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736483 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736485 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736499 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736502 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736505 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736507 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736510 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736512 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736515 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736517 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736522 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736526 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736530 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736533 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736535 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736538 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736541 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:30.794843 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736543 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736546 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736548 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736550 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736553 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736556 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736559 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736562 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736564 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736567 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736570 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736572 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736575 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736577 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736580 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736582 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736585 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736588 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736591 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736594 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:30.795741 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736597 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736599 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736602 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736604 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736609 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736612 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736615 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736618 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736620 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736623 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736626 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736629 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736632 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.736634 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737354 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737361 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737364 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737367 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737370 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737374 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:30.796645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737378 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737381 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737383 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737386 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737389 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737392 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737394 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737398 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737401 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737404 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737406 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737409 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737412 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737416 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737419 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737423 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737425 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737428 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737432 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737435 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:30.797753 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737438 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737441 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737444 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737447 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737450 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737453 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737456 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737458 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737461 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737463 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737466 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737468 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737471 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737473 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737476 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737478 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737481 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737483 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737486 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737502 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:30.798637 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737506 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737509 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737512 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737515 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737519 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737522 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737525 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737529 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737532 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737535 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737538 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737540 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737543 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737546 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737548 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737551 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737554 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737556 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737559 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:30.799730 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737561 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737564 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737566 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737569 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737571 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737575 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737579 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737581 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737584 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737587 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737590 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737593 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737595 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737598 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737600 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737603 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737606 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737608 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737610 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737613 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:30.800810 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737616 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737618 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737622 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737624 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737627 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737630 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737633 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737635 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737638 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737640 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737643 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737646 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737648 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.737651 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738055 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738061 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738064 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738068 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738071 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:30.801688 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738074 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738077 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738080 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738083 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738086 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738089 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738092 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738094 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738098 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738100 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738103 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738105 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738108 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738110 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738113 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738116 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738118 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738121 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738124 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738126 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:30.802760 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738129 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738132 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738134 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738137 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738139 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738142 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738145 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738147 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738150 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738152 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738155 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738157 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738160 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738163 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738165 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738168 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738170 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738173 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738176 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738179 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:30.803645 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738181 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738183 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738186 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738188 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738191 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738194 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738197 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738200 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738203 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738205 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738207 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738210 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738212 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738215 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738219 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738221 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738225 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738229 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738233 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:30.804520 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738236 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738240 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738243 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738245 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738248 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738251 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738254 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738256 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738259 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738261 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738264 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738266 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738269 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738271 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738274 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738277 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738279 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738282 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738284 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738287 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:30.805377 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738290 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738292 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738295 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738298 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738301 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738304 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738308 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738311 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738314 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738316 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738319 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738322 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738324 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738326 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.738329 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738392 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738398 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738405 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738409 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738414 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 22:53:30.806248 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738418 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738422 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738426 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738429 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738432 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738439 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738442 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738445 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738449 2574 flags.go:64] FLAG: --cgroup-root="" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738451 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738454 2574 flags.go:64] FLAG: --client-ca-file="" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738457 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738460 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738464 2574 flags.go:64] FLAG: --cluster-domain="" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738471 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738474 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738477 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738481 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738484 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738500 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738504 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738508 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738511 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738514 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 22:53:30.807127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738517 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738520 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738522 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738526 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738529 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738532 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738535 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738538 2574 flags.go:64] FLAG: --enable-server="true" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738542 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738547 2574 flags.go:64] FLAG: --event-burst="100" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738550 2574 flags.go:64] FLAG: --event-qps="50" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738553 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738556 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738560 2574 flags.go:64] FLAG: --eviction-hard="" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738564 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738567 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738571 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738573 2574 flags.go:64] FLAG: --eviction-soft="" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738577 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738580 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738583 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738585 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738589 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738592 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738595 2574 flags.go:64] FLAG: --feature-gates="" Jul 20 22:53:30.808208 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738599 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738602 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738605 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738608 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738611 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738614 2574 flags.go:64] FLAG: --help="false" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738617 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-134-37.ec2.internal" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738620 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738623 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738625 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738629 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738632 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738635 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738638 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738640 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738643 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738647 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738650 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738653 2574 flags.go:64] FLAG: --kube-reserved="" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738656 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738660 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738663 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738666 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738669 2574 flags.go:64] FLAG: --lock-file="" Jul 20 22:53:30.809533 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738671 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738674 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738677 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738683 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738685 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738689 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738692 2574 flags.go:64] FLAG: --logging-format="text" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738695 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738698 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738701 2574 flags.go:64] FLAG: --manifest-url="" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738711 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738716 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738719 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738723 2574 flags.go:64] FLAG: --max-pods="110" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738725 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738728 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738731 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738734 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738737 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738740 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738743 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738750 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738753 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738756 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 22:53:30.812978 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738759 2574 flags.go:64] FLAG: --pod-cidr="" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738762 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738768 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738771 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738774 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738778 2574 flags.go:64] FLAG: --port="10250" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738781 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738785 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1c0b3ad97ea805f" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738788 2574 flags.go:64] FLAG: --qos-reserved="" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738791 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738794 2574 flags.go:64] FLAG: --register-node="true" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738797 2574 flags.go:64] FLAG: --register-with-taints="" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738800 2574 flags.go:64] FLAG: --registry-burst="10" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738805 2574 flags.go:64] FLAG: --registry-qps="5" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738808 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738811 2574 flags.go:64] FLAG: --reserved-memory="" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738814 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738818 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738821 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738823 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738826 2574 flags.go:64] FLAG: --runonce="false" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738829 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738832 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738835 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738838 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 22:53:30.814089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738840 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738843 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738846 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738849 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738852 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738855 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738857 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738860 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738863 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738867 2574 flags.go:64] FLAG: --system-cgroups="" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738870 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738875 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738879 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738882 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738886 2574 flags.go:64] FLAG: --tls-min-version="" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738889 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738892 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738894 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738897 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738900 2574 flags.go:64] FLAG: --v="2" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738906 2574 flags.go:64] FLAG: --version="false" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738910 2574 flags.go:64] FLAG: --vmodule="" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738913 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.738916 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739031 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:30.815825 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739036 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739040 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739043 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739046 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739049 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739052 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739054 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739057 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739060 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739062 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739065 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739068 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739070 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739073 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739075 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739078 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739081 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739084 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739087 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739090 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:30.816933 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739093 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739096 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739098 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739101 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739104 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739106 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739109 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739113 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739115 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739118 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739120 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739123 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739126 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739128 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739131 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739133 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739136 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739138 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739141 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739144 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:30.817991 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739146 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739149 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739151 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739154 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739157 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739159 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739162 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739164 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739167 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739169 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739172 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739176 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739193 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739197 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739200 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739204 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739206 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739209 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739212 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739216 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:30.818882 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739219 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739221 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739224 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739228 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739231 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739234 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739237 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739241 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739244 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739248 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739250 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739253 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739257 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739260 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739263 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739266 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739269 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739271 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:30.919157 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739275 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:30.833660 ip-10-0-134-37 systemd[1]: Started Kubernetes Kubelet. Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739277 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739280 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739283 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739286 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739290 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739293 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739296 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739298 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739301 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739303 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739306 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739309 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739312 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739315 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739317 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739320 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739322 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739325 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:30.920169 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.739328 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.739945 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.746395 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.746409 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.746428 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.746434 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746502 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746508 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746511 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746515 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746518 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746521 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746524 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746527 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746530 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:30.921797 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746533 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746536 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746538 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746541 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746544 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746546 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746549 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746551 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746554 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746557 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746559 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746562 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746564 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746567 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746569 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746572 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746575 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746578 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746580 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746583 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:30.922856 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746586 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746589 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746591 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746595 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746597 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746600 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746602 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746605 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746611 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746613 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746616 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746619 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746621 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746624 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746627 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746629 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746633 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746635 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746638 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746641 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:30.924138 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746644 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746646 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746649 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746651 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746654 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746657 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746661 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746664 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746667 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746671 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746674 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746678 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746680 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746684 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746686 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746689 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746692 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746694 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746697 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:30.928441 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746700 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746703 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746706 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746709 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746712 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746714 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746717 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746720 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746722 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746725 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746727 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746730 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746733 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746735 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746738 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746741 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746743 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746746 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746750 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:30.929560 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746754 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746757 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746760 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746763 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746765 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746768 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746772 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746775 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746778 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746781 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746784 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746787 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.746792 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746902 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746908 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 22:53:30.930433 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746911 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746914 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746917 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746919 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746922 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746925 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746928 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746931 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746934 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746937 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746940 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746943 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746945 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746950 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746953 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746956 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746959 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746961 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746964 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746967 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 22:53:30.932158 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746969 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746972 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746974 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746978 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746980 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746983 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746987 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746991 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746995 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.746997 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747000 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747005 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747007 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747010 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747012 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747015 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747018 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747020 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 22:53:30.933254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747023 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747026 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747028 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747031 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747033 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747036 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747038 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747042 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747045 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747047 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747050 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747052 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747055 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747057 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747060 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747063 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747065 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747068 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747070 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747073 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747075 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 22:53:30.934116 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747079 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747081 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747084 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747086 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747090 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747093 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747095 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747098 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747101 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747103 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747106 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747108 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747111 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747114 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747116 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747118 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747122 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747126 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747131 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 22:53:30.936846 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747134 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747137 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747140 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747142 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747145 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747149 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747151 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747154 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747156 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747159 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747161 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747164 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747166 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747169 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747172 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747175 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747177 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747183 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 22:53:30.937909 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:30.747185 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.747190 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.747287 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.749473 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.750724 2574 server.go:1019] "Starting client certificate rotation" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.750825 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.750863 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.752942 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.754018 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.765762 2574 log.go:25] "Validated CRI v1 runtime API" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.768288 2574 log.go:25] "Validated CRI v1 image API" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.769010 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 22:53:30.939394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.772768 2574 fs.go:135] Filesystem UUIDs: map[0d113787-88ac-44c4-9feb-b3d31f485394:/dev/nvme0n1p3 2627b544-0d2f-4d88-9868-ec154d26b795:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 20 22:53:30.940048 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.772783 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.786102 2574 manager.go:216] Machine: {Timestamp:2026-07-20 22:53:30.783614109 +0000 UTC m=+0.280068523 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100163 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cd94c72676cc46075a32397e731d0 SystemUUID:ec2cd94c-7267-6cc4-6075-a32397e731d0 BootID:d8db9e9b-9866-4794-bddb-38e32800e2ca Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9f:04:36:21:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9f:04:36:21:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:9f:80:ad:97:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.786756 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.787023 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.788932 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.788954 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.789650 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.789659 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 22:53:30.941153 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.789666 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.790885 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.792246 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.796804 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.796907 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.797542 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.797552 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.797563 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.797572 2574 kubelet.go:398] "Adding apiserver pod source" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.797582 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.797915 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.797929 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.825343 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.827655 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.827685 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828066 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828830 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828844 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828850 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828855 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828864 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828869 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828877 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828883 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828895 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828901 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828909 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.828926 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.829618 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.829624 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.832700 2574 server.go:1268] "Started kubelet" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.833031 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.832957 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.833178 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.836726 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.837909 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cmx7s" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.839001 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.839145 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.840376 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.842523 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cmx7s" Jul 20 22:53:30.941813 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.843228 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.843472 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.844139 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.844151 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.844231 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.843040 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-37.ec2.internal.18c421573819f94a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-37.ec2.internal,UID:ip-10-0-134-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-37.ec2.internal,},FirstTimestamp:2026-07-20 22:53:30.832656714 +0000 UTC m=+0.329111195,LastTimestamp:2026-07-20 22:53:30.832656714 +0000 UTC m=+0.329111195,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-37.ec2.internal,}" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.844667 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.844676 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.844705 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.844841 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.844849 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.849698 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.855794 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.855813 2574 factory.go:55] Registering systemd factory Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.855828 2574 factory.go:223] Registration of the systemd container factory successfully Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.856051 2574 factory.go:153] Registering CRI-O factory Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.856066 2574 factory.go:223] Registration of the crio container factory successfully Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.856124 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.856149 2574 factory.go:103] Registering Raw factory Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.856166 2574 manager.go:1196] Started watching for new ooms in manager Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.856516 2574 manager.go:349] Starting recovery of all containers Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.866056 2574 manager.go:354] Recovery completed Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.875408 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.877268 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.877301 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.877313 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.877846 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.877857 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.877871 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.877896 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.878610 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:30.944800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.878633 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.878642 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.879748 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-37.ec2.internal.18c421573ac2f157 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-37.ec2.internal,UID:ip-10-0-134-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-37.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-37.ec2.internal,},FirstTimestamp:2026-07-20 22:53:30.877284695 +0000 UTC m=+0.373739110,LastTimestamp:2026-07-20 22:53:30.877284695 +0000 UTC m=+0.373739110,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-37.ec2.internal,}" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.879993 2574 policy_none.go:49] "None policy: Start" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.880009 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.880022 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.882062 2574 policy_none.go:47] "Start" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.944731 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.960099 2574 manager.go:337] "Starting Device Plugin manager" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.960128 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.960136 2574 server.go:87] "Starting device plugin registration server" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.960367 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.960379 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.960549 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.960629 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:30.960637 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.962233 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 22:53:30.966925 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:30.962277 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.001439 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.002567 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.002604 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.002624 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.002656 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.005249 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.059435 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-37.ec2.internal\" not found" node="ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.060531 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.062986 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.063018 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.063031 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.063055 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.072544 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.072559 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-37.ec2.internal\": node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.098566 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.103648 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal"] Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.103699 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.104402 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.104426 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.104441 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107005 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107140 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107163 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107685 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107692 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107708 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107712 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107722 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.107726 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.110252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.110278 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.112223 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientMemory" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.112243 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 22:53:31.128089 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.112253 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeHasSufficientPID" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.143288 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-37.ec2.internal\" not found" node="ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.145744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40387f4cb1b49043baacd9f958633c93-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.ec2.internal\" (UID: \"40387f4cb1b49043baacd9f958633c93\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.145769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09139888c4ca3c1b45f388babb2f2bd5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal\" (UID: \"09139888c4ca3c1b45f388babb2f2bd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.145792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09139888c4ca3c1b45f388babb2f2bd5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal\" (UID: \"09139888c4ca3c1b45f388babb2f2bd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.147761 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-37.ec2.internal\" not found" node="ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.199406 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.247247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40387f4cb1b49043baacd9f958633c93-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.ec2.internal\" (UID: \"40387f4cb1b49043baacd9f958633c93\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.247298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09139888c4ca3c1b45f388babb2f2bd5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal\" (UID: \"09139888c4ca3c1b45f388babb2f2bd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.247317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09139888c4ca3c1b45f388babb2f2bd5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal\" (UID: \"09139888c4ca3c1b45f388babb2f2bd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.247335 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40387f4cb1b49043baacd9f958633c93-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.ec2.internal\" (UID: \"40387f4cb1b49043baacd9f958633c93\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.247351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09139888c4ca3c1b45f388babb2f2bd5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal\" (UID: \"09139888c4ca3c1b45f388babb2f2bd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.247356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09139888c4ca3c1b45f388babb2f2bd5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal\" (UID: \"09139888c4ca3c1b45f388babb2f2bd5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.317652 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.302898 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.513951 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.407095 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.513951 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.447314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.513951 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.450751 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.513951 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.507683 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.701034 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.611981 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.712047 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-37.ec2.internal\" not found" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.730245 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.744377 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.749888 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.749999 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.750031 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.750073 2574 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a87cd87b13e704353ab68ffe89a899eb-720addcafdf0fd55.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.134.37:49740->3.95.94.238:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.750104 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.767312 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.800044 2574 apiserver.go:52] "Watching apiserver" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.808120 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.808887 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zjsrr","openshift-multus/multus-additional-cni-plugins-bsqhr","openshift-network-diagnostics/network-check-target-88x7j","openshift-network-operator/iptables-alerter-ghqts","kube-system/global-pull-secret-syncer-xl88f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b","openshift-image-registry/node-ca-vwqjp","openshift-multus/multus-b6lr4","openshift-multus/network-metrics-daemon-gw7vz","openshift-ovn-kubernetes/ovnkube-node-sg4m5","kube-system/konnectivity-agent-nnwwg","kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tsrlf"] Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.811743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.813178 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.814199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.814545 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.814737 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n9fvb\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.815244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.815413 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.818236 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.818461 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.818644 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.818912 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.819104 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.819262 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thj8l\"" Jul 20 22:53:31.830463 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.819435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.819488 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.819716 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.821510 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.821890 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tcvf4\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.822083 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.822307 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.822484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.822531 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.823828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.825939 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.826110 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ft5lv\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.826341 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.826517 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.826903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.828745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.829188 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.829378 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nnr6s\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.830005 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.830802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.830850 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.831061 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 22:53:31.832801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.831234 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ft7p5\"" Jul 20 22:53:31.835816 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.835183 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.837519 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.837166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:31.838101 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.837654 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 22:53:31.838101 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.837681 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 22:53:31.838101 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.837706 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pkmkh\"" Jul 20 22:53:31.838101 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.838049 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 22:53:31.838101 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.838063 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.838203 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.838207 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.839194 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.839294 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z2ppv\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.840215 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.840301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.843460 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.843468 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8564l\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.843460 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.844329 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.845599 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-cnibin\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-cni-bin\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqln\" (UniqueName: \"kubernetes.io/projected/4532e5bf-184a-4cb3-b583-8451ff8a014e-kube-api-access-bvqln\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c0a5da1-61a9-4426-9646-800e40d3482c-serviceca\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850284 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-os-release\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.971119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-k8s-cni-cncf-io\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-etc-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-var-lib-kubelet\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-cni-bin\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb100de-3890-47c2-868d-79a783d51b47-tmp-dir\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysctl-conf\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.850525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-sys\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.854500 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-kubelet-config\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkk8\" (UniqueName: \"kubernetes.io/projected/b1678de8-3d36-424d-b897-81373118ab57-kube-api-access-flkk8\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-os-release\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-systemd\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.972263 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-dbus\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvqwk\" (UniqueName: \"kubernetes.io/projected/2d4e33a0-0442-47b0-9ecf-e833f909a185-kube-api-access-nvqwk\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-host\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-device-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-sys-fs\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-cni-multus\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-etc-kubernetes\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-var-lib-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-cni-netd\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-modprobe-d\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-kubernetes\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856734 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbba1515-089c-438d-8ba9-56dd9d35f6b1-cni-binary-copy\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856763 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-log-socket\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.975274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4532e5bf-184a-4cb3-b583-8451ff8a014e-tmp\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-run-netns\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d5f7832-f567-42b5-966f-4084d961c60b-agent-certs\") pod \"konnectivity-agent-nnwwg\" (UID: \"4d5f7832-f567-42b5-966f-4084d961c60b\") " pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-slash\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/feb100de-3890-47c2-868d-79a783d51b47-hosts-file\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.856983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-kubelet\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdpc\" (UniqueName: \"kubernetes.io/projected/dbba1515-089c-438d-8ba9-56dd9d35f6b1-kube-api-access-9kdpc\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857093 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-cnibin\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhrx\" (UniqueName: \"kubernetes.io/projected/ac8deb1c-423a-4303-95a7-f9a8fb23a892-kube-api-access-fzhrx\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmmdg\" (UniqueName: \"kubernetes.io/projected/feb100de-3890-47c2-868d-79a783d51b47-kube-api-access-lmmdg\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-system-cni-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.976915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-netns\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-env-overrides\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-daemon-config\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0a5da1-61a9-4426-9646-800e40d3482c-host\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-cni-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857277 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 22:48:31 +0000 UTC" deadline="2028-03-04 23:17:04.636192161 +0000 UTC" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-ovnkube-script-lib\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857309 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14232h23m32.778885763s" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysconfig\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac8deb1c-423a-4303-95a7-f9a8fb23a892-host-slash\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-socket-dir-parent\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-kubelet\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-run\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcnj8\" (UniqueName: \"kubernetes.io/projected/6c0a5da1-61a9-4426-9646-800e40d3482c-kube-api-access-lcnj8\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-system-cni-dir\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.980349 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac8deb1c-423a-4303-95a7-f9a8fb23a892-iptables-alerter-script\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-conf-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-systemd\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxq6\" (UniqueName: \"kubernetes.io/projected/9c326231-6e02-4973-aa69-116110666151-kube-api-access-mjxq6\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-multus-certs\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-systemd-units\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-ovn\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysctl-d\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-tuned\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857734 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-socket-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6caa92d9-60c0-4340-8426-2f60704881bf-ovn-node-metrics-cert\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-hostroot\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.981271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-node-log\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-ovnkube-config\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2n85\" (UniqueName: \"kubernetes.io/projected/6caa92d9-60c0-4340-8426-2f60704881bf-kube-api-access-t2n85\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d5f7832-f567-42b5-966f-4084d961c60b-konnectivity-ca\") pod \"konnectivity-agent-nnwwg\" (UID: \"4d5f7832-f567-42b5-966f-4084d961c60b\") " pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-lib-modules\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-registration-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.857931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.881852 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-crsg9" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.888682 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-crsg9" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-dbus\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqwk\" (UniqueName: \"kubernetes.io/projected/2d4e33a0-0442-47b0-9ecf-e833f909a185-kube-api-access-nvqwk\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-host\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-device-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-sys-fs\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-cni-multus\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.983546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-etc-kubernetes\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-etc-kubernetes\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-dbus\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958736 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-cni-multus\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-device-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-sys-fs\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958720 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-host\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-var-lib-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-cni-netd\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-modprobe-d\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-kubernetes\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958834 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-var-lib-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-cni-netd\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbba1515-089c-438d-8ba9-56dd9d35f6b1-cni-binary-copy\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-log-socket\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984098 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-kubernetes\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-modprobe-d\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4532e5bf-184a-4cb3-b583-8451ff8a014e-tmp\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-log-socket\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-run-netns\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.958996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d5f7832-f567-42b5-966f-4084d961c60b-agent-certs\") pod \"konnectivity-agent-nnwwg\" (UID: \"4d5f7832-f567-42b5-966f-4084d961c60b\") " pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-slash\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.959056 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-slash\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-run-netns\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.959147 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs podName:2d4e33a0-0442-47b0-9ecf-e833f909a185 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:32.459106597 +0000 UTC m=+1.955561018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs") pod "network-metrics-daemon-gw7vz" (UID: "2d4e33a0-0442-47b0-9ecf-e833f909a185") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/feb100de-3890-47c2-868d-79a783d51b47-hosts-file\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-kubelet\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.984832 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959261 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdpc\" (UniqueName: \"kubernetes.io/projected/dbba1515-089c-438d-8ba9-56dd9d35f6b1-kube-api-access-9kdpc\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-cnibin\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-kubelet\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/feb100de-3890-47c2-868d-79a783d51b47-hosts-file\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-cnibin\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbba1515-089c-438d-8ba9-56dd9d35f6b1-cni-binary-copy\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhrx\" (UniqueName: \"kubernetes.io/projected/ac8deb1c-423a-4303-95a7-f9a8fb23a892-kube-api-access-fzhrx\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmdg\" (UniqueName: \"kubernetes.io/projected/feb100de-3890-47c2-868d-79a783d51b47-kube-api-access-lmmdg\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-system-cni-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-netns\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-env-overrides\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-system-cni-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.985693 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-netns\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-daemon-config\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0a5da1-61a9-4426-9646-800e40d3482c-host\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-cni-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-ovnkube-script-lib\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0a5da1-61a9-4426-9646-800e40d3482c-host\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysconfig\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac8deb1c-423a-4303-95a7-f9a8fb23a892-host-slash\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-socket-dir-parent\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysconfig\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-kubelet\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-cni-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-run\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959906 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-run\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959934 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.986542 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959943 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac8deb1c-423a-4303-95a7-f9a8fb23a892-host-slash\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-kubelet\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959997 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-socket-dir-parent\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.959902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcnj8\" (UniqueName: \"kubernetes.io/projected/6c0a5da1-61a9-4426-9646-800e40d3482c-kube-api-access-lcnj8\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-system-cni-dir\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac8deb1c-423a-4303-95a7-f9a8fb23a892-iptables-alerter-script\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-conf-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-system-cni-dir\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-conf-dir\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-systemd\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbba1515-089c-438d-8ba9-56dd9d35f6b1-multus-daemon-config\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxq6\" (UniqueName: \"kubernetes.io/projected/9c326231-6e02-4973-aa69-116110666151-kube-api-access-mjxq6\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-env-overrides\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-multus-certs\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.987270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-systemd\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-systemd-units\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960231 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-multus-certs\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-ovn\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-systemd-units\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysctl-d\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-ovnkube-script-lib\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960331 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-ovn\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-tuned\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-socket-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960447 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysctl-d\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6caa92d9-60c0-4340-8426-2f60704881bf-ovn-node-metrics-cert\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.990489 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-socket-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960589 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac8deb1c-423a-4303-95a7-f9a8fb23a892-iptables-alerter-script\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-hostroot\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-node-log\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-ovnkube-config\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2n85\" (UniqueName: \"kubernetes.io/projected/6caa92d9-60c0-4340-8426-2f60704881bf-kube-api-access-t2n85\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-node-log\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d5f7832-f567-42b5-966f-4084d961c60b-konnectivity-ca\") pod \"konnectivity-agent-nnwwg\" (UID: \"4d5f7832-f567-42b5-966f-4084d961c60b\") " pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-hostroot\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-lib-modules\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-registration-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.991158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960960 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-lib-modules\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.960985 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-registration-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-cnibin\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-cnibin\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.961139 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.961204 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret podName:ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:32.461183337 +0000 UTC m=+1.957637756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret") pod "global-pull-secret-syncer-xl88f" (UID: "ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-cni-bin\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqln\" (UniqueName: \"kubernetes.io/projected/4532e5bf-184a-4cb3-b583-8451ff8a014e-kube-api-access-bvqln\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4d5f7832-f567-42b5-966f-4084d961c60b-konnectivity-ca\") pod \"konnectivity-agent-nnwwg\" (UID: \"4d5f7832-f567-42b5-966f-4084d961c60b\") " pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c0a5da1-61a9-4426-9646-800e40d3482c-serviceca\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:31.991837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-cni-bin\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-os-release\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-k8s-cni-cncf-io\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-etc-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961524 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-var-lib-kubelet\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-cni-bin\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb100de-3890-47c2-868d-79a783d51b47-tmp-dir\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961607 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysctl-conf\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1678de8-3d36-424d-b897-81373118ab57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-sys\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-kubelet-config\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:32.114919 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961714 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c326231-6e02-4973-aa69-116110666151-os-release\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkk8\" (UniqueName: \"kubernetes.io/projected/b1678de8-3d36-424d-b897-81373118ab57-kube-api-access-flkk8\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961747 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c0a5da1-61a9-4426-9646-800e40d3482c-serviceca\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-sysctl-conf\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-os-release\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-kubelet-config\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-run-k8s-cni-cncf-io\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-host-var-lib-cni-bin\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961976 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-etc-openvswitch\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-sys\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962035 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4532e5bf-184a-4cb3-b583-8451ff8a014e-var-lib-kubelet\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.961751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbba1515-089c-438d-8ba9-56dd9d35f6b1-os-release\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-systemd\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962164 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.116004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6caa92d9-60c0-4340-8426-2f60704881bf-run-systemd\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb100de-3890-47c2-868d-79a783d51b47-tmp-dir\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c326231-6e02-4973-aa69-116110666151-cni-binary-copy\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.962708 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6caa92d9-60c0-4340-8426-2f60704881bf-ovnkube-config\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.963160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4532e5bf-184a-4cb3-b583-8451ff8a014e-etc-tuned\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.963166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6caa92d9-60c0-4340-8426-2f60704881bf-ovn-node-metrics-cert\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.963224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4532e5bf-184a-4cb3-b583-8451ff8a014e-tmp\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.963374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4d5f7832-f567-42b5-966f-4084d961c60b-agent-certs\") pod \"konnectivity-agent-nnwwg\" (UID: \"4d5f7832-f567-42b5-966f-4084d961c60b\") " pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.971992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvqwk\" (UniqueName: \"kubernetes.io/projected/2d4e33a0-0442-47b0-9ecf-e833f909a185-kube-api-access-nvqwk\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.974548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdpc\" (UniqueName: \"kubernetes.io/projected/dbba1515-089c-438d-8ba9-56dd9d35f6b1-kube-api-access-9kdpc\") pod \"multus-b6lr4\" (UID: \"dbba1515-089c-438d-8ba9-56dd9d35f6b1\") " pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.974736 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.974758 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.974772 2574 projected.go:196] Error preparing data for projected volume kube-api-access-wxv8w for pod openshift-network-diagnostics/network-check-target-88x7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:31.974835 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w podName:fcc384bb-5ccb-4609-948b-88c41d22bb2a nodeName:}" failed. No retries permitted until 2026-07-20 22:53:32.474818617 +0000 UTC m=+1.971273042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wxv8w" (UniqueName: "kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w") pod "network-check-target-88x7j" (UID: "fcc384bb-5ccb-4609-948b-88c41d22bb2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:32.116870 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.975338 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhrx\" (UniqueName: \"kubernetes.io/projected/ac8deb1c-423a-4303-95a7-f9a8fb23a892-kube-api-access-fzhrx\") pod \"iptables-alerter-ghqts\" (UID: \"ac8deb1c-423a-4303-95a7-f9a8fb23a892\") " pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.975396 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcnj8\" (UniqueName: \"kubernetes.io/projected/6c0a5da1-61a9-4426-9646-800e40d3482c-kube-api-access-lcnj8\") pod \"node-ca-vwqjp\" (UID: \"6c0a5da1-61a9-4426-9646-800e40d3482c\") " pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.975940 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxq6\" (UniqueName: \"kubernetes.io/projected/9c326231-6e02-4973-aa69-116110666151-kube-api-access-mjxq6\") pod \"multus-additional-cni-plugins-bsqhr\" (UID: \"9c326231-6e02-4973-aa69-116110666151\") " pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.977885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2n85\" (UniqueName: \"kubernetes.io/projected/6caa92d9-60c0-4340-8426-2f60704881bf-kube-api-access-t2n85\") pod \"ovnkube-node-sg4m5\" (UID: \"6caa92d9-60c0-4340-8426-2f60704881bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.978028 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqln\" (UniqueName: \"kubernetes.io/projected/4532e5bf-184a-4cb3-b583-8451ff8a014e-kube-api-access-bvqln\") pod \"tuned-tsrlf\" (UID: \"4532e5bf-184a-4cb3-b583-8451ff8a014e\") " pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.978066 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmdg\" (UniqueName: \"kubernetes.io/projected/feb100de-3890-47c2-868d-79a783d51b47-kube-api-access-lmmdg\") pod \"node-resolver-zjsrr\" (UID: \"feb100de-3890-47c2-868d-79a783d51b47\") " pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.978525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkk8\" (UniqueName: \"kubernetes.io/projected/b1678de8-3d36-424d-b897-81373118ab57-kube-api-access-flkk8\") pod \"aws-ebs-csi-driver-node-rt55b\" (UID: \"b1678de8-3d36-424d-b897-81373118ab57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:31.998396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.020180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:32.118608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.024218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" Jul 20 22:53:32.195938 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.153890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vwqjp" Jul 20 22:53:32.195938 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.180727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" Jul 20 22:53:32.195938 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.187555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghqts" Jul 20 22:53:32.215673 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.215651 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" Jul 20 22:53:32.239706 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.239683 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjsrr" Jul 20 22:53:32.245590 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.245570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6lr4" Jul 20 22:53:32.290652 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.290625 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40387f4cb1b49043baacd9f958633c93.slice/crio-55bb11aaab7a22470795a7c4d25fe5b3f34a91ffecf2b25c9ec5ce10a0f50ce0 WatchSource:0}: Error finding container 55bb11aaab7a22470795a7c4d25fe5b3f34a91ffecf2b25c9ec5ce10a0f50ce0: Status 404 returned error can't find the container with id 55bb11aaab7a22470795a7c4d25fe5b3f34a91ffecf2b25c9ec5ce10a0f50ce0 Jul 20 22:53:32.291254 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.291227 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1678de8_3d36_424d_b897_81373118ab57.slice/crio-86eb4735d3c44c98507e0dd443a90a91054234258de1f086e8e562f676ef0c4a WatchSource:0}: Error finding container 86eb4735d3c44c98507e0dd443a90a91054234258de1f086e8e562f676ef0c4a: Status 404 returned error can't find the container with id 86eb4735d3c44c98507e0dd443a90a91054234258de1f086e8e562f676ef0c4a Jul 20 22:53:32.291903 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.291879 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09139888c4ca3c1b45f388babb2f2bd5.slice/crio-0a5433634d84b69958fe2cba29f3276e523c4dcea46d4b0f4531b5186b774036 WatchSource:0}: Error finding container 0a5433634d84b69958fe2cba29f3276e523c4dcea46d4b0f4531b5186b774036: Status 404 returned error can't find the container with id 0a5433634d84b69958fe2cba29f3276e523c4dcea46d4b0f4531b5186b774036 Jul 20 22:53:32.292974 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.292951 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbba1515_089c_438d_8ba9_56dd9d35f6b1.slice/crio-12f055ff2a9b2790d08c5b8a4f47621c9dad51140314de6eb0f4025d0387739d WatchSource:0}: Error finding container 12f055ff2a9b2790d08c5b8a4f47621c9dad51140314de6eb0f4025d0387739d: Status 404 returned error can't find the container with id 12f055ff2a9b2790d08c5b8a4f47621c9dad51140314de6eb0f4025d0387739d Jul 20 22:53:32.300212 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.300196 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 22:53:32.395985 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.395918 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:32.398792 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.398769 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5f7832_f567_42b5_966f_4084d961c60b.slice/crio-5ce8f42308352b125543318d5bb52ce7cf3bdb71a3d6b9d6324cedaa5d6fe1c0 WatchSource:0}: Error finding container 5ce8f42308352b125543318d5bb52ce7cf3bdb71a3d6b9d6324cedaa5d6fe1c0: Status 404 returned error can't find the container with id 5ce8f42308352b125543318d5bb52ce7cf3bdb71a3d6b9d6324cedaa5d6fe1c0 Jul 20 22:53:32.466940 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.466914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:32.467088 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.466974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:32.467088 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.467064 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:32.467217 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.467123 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs podName:2d4e33a0-0442-47b0-9ecf-e833f909a185 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:33.467104331 +0000 UTC m=+2.963558734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs") pod "network-metrics-daemon-gw7vz" (UID: "2d4e33a0-0442-47b0-9ecf-e833f909a185") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:32.467217 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.467136 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:32.467217 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.467192 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret podName:ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:33.46717287 +0000 UTC m=+2.963627274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret") pod "global-pull-secret-syncer-xl88f" (UID: "ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:32.473468 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.473439 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6caa92d9_60c0_4340_8426_2f60704881bf.slice/crio-072ebb1766f32ad0f4f5208ed10f5c9db8daa5e86b1cad06660644b62f7506d8 WatchSource:0}: Error finding container 072ebb1766f32ad0f4f5208ed10f5c9db8daa5e86b1cad06660644b62f7506d8: Status 404 returned error can't find the container with id 072ebb1766f32ad0f4f5208ed10f5c9db8daa5e86b1cad06660644b62f7506d8 Jul 20 22:53:32.567302 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.567267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:32.567446 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.567428 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:32.567446 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.567446 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:32.567592 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.567457 2574 projected.go:196] Error preparing data for projected volume kube-api-access-wxv8w for pod openshift-network-diagnostics/network-check-target-88x7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:32.567592 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:32.567499 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w podName:fcc384bb-5ccb-4609-948b-88c41d22bb2a nodeName:}" failed. No retries permitted until 2026-07-20 22:53:33.567483411 +0000 UTC m=+3.063937815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxv8w" (UniqueName: "kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w") pod "network-check-target-88x7j" (UID: "fcc384bb-5ccb-4609-948b-88c41d22bb2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:32.580268 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.580241 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8deb1c_423a_4303_95a7_f9a8fb23a892.slice/crio-f92abfeb0bdf232d821ffb5c3ffea850c1c43b2897772adb90d284fcd1292509 WatchSource:0}: Error finding container f92abfeb0bdf232d821ffb5c3ffea850c1c43b2897772adb90d284fcd1292509: Status 404 returned error can't find the container with id f92abfeb0bdf232d821ffb5c3ffea850c1c43b2897772adb90d284fcd1292509 Jul 20 22:53:32.617098 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.617065 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0a5da1_61a9_4426_9646_800e40d3482c.slice/crio-7fd7e9ab4914821857f71c93bd756a210473910119e2fb824ea757567659df90 WatchSource:0}: Error finding container 7fd7e9ab4914821857f71c93bd756a210473910119e2fb824ea757567659df90: Status 404 returned error can't find the container with id 7fd7e9ab4914821857f71c93bd756a210473910119e2fb824ea757567659df90 Jul 20 22:53:32.625891 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.625866 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4532e5bf_184a_4cb3_b583_8451ff8a014e.slice/crio-e082bc69b0d5f95072d15fe197687088b825b17a3ca4d07fd7affa16d8410f91 WatchSource:0}: Error finding container e082bc69b0d5f95072d15fe197687088b825b17a3ca4d07fd7affa16d8410f91: Status 404 returned error can't find the container with id e082bc69b0d5f95072d15fe197687088b825b17a3ca4d07fd7affa16d8410f91 Jul 20 22:53:32.636873 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.636852 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c326231_6e02_4973_aa69_116110666151.slice/crio-38962cd7fe67feb866f22ae39284b9291fb766a5eb78f4878b8c8dd5bdeea8ce WatchSource:0}: Error finding container 38962cd7fe67feb866f22ae39284b9291fb766a5eb78f4878b8c8dd5bdeea8ce: Status 404 returned error can't find the container with id 38962cd7fe67feb866f22ae39284b9291fb766a5eb78f4878b8c8dd5bdeea8ce Jul 20 22:53:32.760885 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:53:32.760838 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb100de_3890_47c2_868d_79a783d51b47.slice/crio-73b277229ce24d0ef0c027207695d567727b3c50f64e39b40bb2b54ba1faaf90 WatchSource:0}: Error finding container 73b277229ce24d0ef0c027207695d567727b3c50f64e39b40bb2b54ba1faaf90: Status 404 returned error can't find the container with id 73b277229ce24d0ef0c027207695d567727b3c50f64e39b40bb2b54ba1faaf90 Jul 20 22:53:32.892224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.892092 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 22:48:32 +0000 UTC" deadline="2028-01-28 22:03:32.120201533 +0000 UTC" Jul 20 22:53:32.892224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:32.892169 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13367h9m59.228036447s" Jul 20 22:53:33.007242 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.006358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:33.007242 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.006494 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:33.007242 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.006926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:33.007242 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.007023 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:33.007242 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.007092 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:33.007242 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.007177 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:33.012075 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.011987 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" event={"ID":"09139888c4ca3c1b45f388babb2f2bd5","Type":"ContainerStarted","Data":"0a5433634d84b69958fe2cba29f3276e523c4dcea46d4b0f4531b5186b774036"} Jul 20 22:53:33.014110 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.014031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6lr4" event={"ID":"dbba1515-089c-438d-8ba9-56dd9d35f6b1","Type":"ContainerStarted","Data":"12f055ff2a9b2790d08c5b8a4f47621c9dad51140314de6eb0f4025d0387739d"} Jul 20 22:53:33.016184 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.016153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" event={"ID":"b1678de8-3d36-424d-b897-81373118ab57","Type":"ContainerStarted","Data":"86eb4735d3c44c98507e0dd443a90a91054234258de1f086e8e562f676ef0c4a"} Jul 20 22:53:33.017716 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.017648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjsrr" event={"ID":"feb100de-3890-47c2-868d-79a783d51b47","Type":"ContainerStarted","Data":"73b277229ce24d0ef0c027207695d567727b3c50f64e39b40bb2b54ba1faaf90"} Jul 20 22:53:33.019399 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.019332 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerStarted","Data":"38962cd7fe67feb866f22ae39284b9291fb766a5eb78f4878b8c8dd5bdeea8ce"} Jul 20 22:53:33.022625 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.022546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vwqjp" event={"ID":"6c0a5da1-61a9-4426-9646-800e40d3482c","Type":"ContainerStarted","Data":"7fd7e9ab4914821857f71c93bd756a210473910119e2fb824ea757567659df90"} Jul 20 22:53:33.024173 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.024146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"072ebb1766f32ad0f4f5208ed10f5c9db8daa5e86b1cad06660644b62f7506d8"} Jul 20 22:53:33.026159 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.026133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nnwwg" event={"ID":"4d5f7832-f567-42b5-966f-4084d961c60b","Type":"ContainerStarted","Data":"5ce8f42308352b125543318d5bb52ce7cf3bdb71a3d6b9d6324cedaa5d6fe1c0"} Jul 20 22:53:33.030662 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.030635 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" event={"ID":"40387f4cb1b49043baacd9f958633c93","Type":"ContainerStarted","Data":"55bb11aaab7a22470795a7c4d25fe5b3f34a91ffecf2b25c9ec5ce10a0f50ce0"} Jul 20 22:53:33.033137 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.033113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" event={"ID":"4532e5bf-184a-4cb3-b583-8451ff8a014e","Type":"ContainerStarted","Data":"e082bc69b0d5f95072d15fe197687088b825b17a3ca4d07fd7affa16d8410f91"} Jul 20 22:53:33.034852 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.034804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghqts" event={"ID":"ac8deb1c-423a-4303-95a7-f9a8fb23a892","Type":"ContainerStarted","Data":"f92abfeb0bdf232d821ffb5c3ffea850c1c43b2897772adb90d284fcd1292509"} Jul 20 22:53:33.275517 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.275255 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:33.474644 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.474056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:33.474644 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.474142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:33.474644 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.474203 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:33.474644 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.474262 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:33.474644 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.474280 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret podName:ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:35.474258467 +0000 UTC m=+4.970712889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret") pod "global-pull-secret-syncer-xl88f" (UID: "ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:33.474644 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.474318 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs podName:2d4e33a0-0442-47b0-9ecf-e833f909a185 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:35.474300056 +0000 UTC m=+4.970754485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs") pod "network-metrics-daemon-gw7vz" (UID: "2d4e33a0-0442-47b0-9ecf-e833f909a185") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:33.575347 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.575172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:33.575554 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.575369 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:33.575554 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.575389 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:33.575554 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.575431 2574 projected.go:196] Error preparing data for projected volume kube-api-access-wxv8w for pod openshift-network-diagnostics/network-check-target-88x7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:33.576109 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:33.576063 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w podName:fcc384bb-5ccb-4609-948b-88c41d22bb2a nodeName:}" failed. No retries permitted until 2026-07-20 22:53:35.575508649 +0000 UTC m=+5.071963055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxv8w" (UniqueName: "kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w") pod "network-check-target-88x7j" (UID: "fcc384bb-5ccb-4609-948b-88c41d22bb2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:33.892815 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.892723 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 22:48:32 +0000 UTC" deadline="2028-02-14 06:06:49.299371222 +0000 UTC" Jul 20 22:53:33.892815 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:33.892765 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13759h13m15.406610443s" Jul 20 22:53:34.283972 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:34.283700 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 22:53:35.002992 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:35.002928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:35.003457 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.003076 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:35.003581 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:35.003524 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:35.003674 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.003651 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:35.003761 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:35.003727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:35.003840 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.003819 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:35.492887 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:35.492849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:35.493090 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:35.492929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:35.493090 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.493080 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:35.493232 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.493143 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret podName:ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:39.493123502 +0000 UTC m=+8.989577907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret") pod "global-pull-secret-syncer-xl88f" (UID: "ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:35.493599 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.493577 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:35.493719 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.493634 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs podName:2d4e33a0-0442-47b0-9ecf-e833f909a185 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:39.493617431 +0000 UTC m=+8.990071852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs") pod "network-metrics-daemon-gw7vz" (UID: "2d4e33a0-0442-47b0-9ecf-e833f909a185") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:35.595056 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:35.594332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:35.595056 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.594584 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:35.595056 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.594609 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:35.595056 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.594624 2574 projected.go:196] Error preparing data for projected volume kube-api-access-wxv8w for pod openshift-network-diagnostics/network-check-target-88x7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:35.595056 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:35.594686 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w podName:fcc384bb-5ccb-4609-948b-88c41d22bb2a nodeName:}" failed. No retries permitted until 2026-07-20 22:53:39.59466573 +0000 UTC m=+9.091120159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxv8w" (UniqueName: "kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w") pod "network-check-target-88x7j" (UID: "fcc384bb-5ccb-4609-948b-88c41d22bb2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:37.003785 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:37.003749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:37.004213 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:37.003879 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:37.004270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:37.004250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:37.004369 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:37.004343 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:37.004723 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:37.004559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:37.004723 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:37.004675 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:39.006923 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:39.006712 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:39.006923 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.006845 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:39.007398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:39.007210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:39.007398 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.007327 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:39.007398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:39.007375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:39.007685 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.007446 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:39.526801 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:39.526762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:39.526994 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:39.526841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:39.526994 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.526965 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:39.527099 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.527044 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret podName:ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:47.527023839 +0000 UTC m=+17.023478248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret") pod "global-pull-secret-syncer-xl88f" (UID: "ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:39.527192 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.526969 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:39.527300 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.527256 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs podName:2d4e33a0-0442-47b0-9ecf-e833f909a185 nodeName:}" failed. No retries permitted until 2026-07-20 22:53:47.527233203 +0000 UTC m=+17.023687620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs") pod "network-metrics-daemon-gw7vz" (UID: "2d4e33a0-0442-47b0-9ecf-e833f909a185") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:39.627664 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:39.627624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:39.627843 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.627804 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:39.627843 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.627832 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:39.627952 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.627857 2574 projected.go:196] Error preparing data for projected volume kube-api-access-wxv8w for pod openshift-network-diagnostics/network-check-target-88x7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:39.627952 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:39.627924 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w podName:fcc384bb-5ccb-4609-948b-88c41d22bb2a nodeName:}" failed. No retries permitted until 2026-07-20 22:53:47.627903689 +0000 UTC m=+17.124358105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxv8w" (UniqueName: "kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w") pod "network-check-target-88x7j" (UID: "fcc384bb-5ccb-4609-948b-88c41d22bb2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:41.004824 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:41.004413 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:41.004824 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:41.004523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:41.004824 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:41.004568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:41.004824 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:41.004661 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:41.005445 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:41.005046 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:41.005445 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:41.005173 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:43.007050 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:43.007012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:43.007515 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:43.007147 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:43.007515 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:43.007511 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:43.007631 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:43.007590 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:43.007631 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:43.007611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:43.007691 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:43.007652 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:45.003700 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:45.003663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:45.004143 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:45.003669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:45.004143 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:45.003795 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:45.004143 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:45.003809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:45.004143 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:45.003887 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:45.004143 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:45.003987 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:47.003192 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:47.003160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:47.003627 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:47.003160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:47.003627 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.003294 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:47.003627 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:47.003160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:47.003627 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.003364 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:47.003627 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.003452 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:47.586856 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:47.586705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:47.586856 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:47.586799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:47.586856 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.586861 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:47.587127 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.586932 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret podName:ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:03.586913425 +0000 UTC m=+33.083367838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret") pod "global-pull-secret-syncer-xl88f" (UID: "ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:53:47.587127 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.586929 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:47.587127 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.586966 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs podName:2d4e33a0-0442-47b0-9ecf-e833f909a185 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:03.58695747 +0000 UTC m=+33.083411871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs") pod "network-metrics-daemon-gw7vz" (UID: "2d4e33a0-0442-47b0-9ecf-e833f909a185") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:53:47.687724 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:47.687685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:47.687911 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.687881 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:53:47.687911 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.687906 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:53:47.688023 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.687921 2574 projected.go:196] Error preparing data for projected volume kube-api-access-wxv8w for pod openshift-network-diagnostics/network-check-target-88x7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:47.688023 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:47.687983 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w podName:fcc384bb-5ccb-4609-948b-88c41d22bb2a nodeName:}" failed. No retries permitted until 2026-07-20 22:54:03.687967739 +0000 UTC m=+33.184422160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxv8w" (UniqueName: "kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w") pod "network-check-target-88x7j" (UID: "fcc384bb-5ccb-4609-948b-88c41d22bb2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:53:49.003750 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:49.003714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:49.004266 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:49.003716 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:49.004266 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:49.003856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:49.004266 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:49.003847 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:49.004266 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:49.003949 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:49.004266 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:49.004067 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:51.003841 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:51.003800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:51.004320 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:51.003922 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:51.004320 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:51.003947 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:51.004320 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:51.004037 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:51.004320 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:51.004072 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:51.004320 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:51.004099 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:53.003704 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.003542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:53.004020 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.003566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:53.004020 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:53.003784 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:53.004020 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.003574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:53.004020 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:53.003882 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:53.004020 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:53.003924 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:53.098287 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.098253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" event={"ID":"40387f4cb1b49043baacd9f958633c93","Type":"ContainerStarted","Data":"fcdd99d1d421f650f31193cfd8813cfad895476e0e5e39e57a4305cc7df75e4e"} Jul 20 22:53:53.099667 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.099639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" event={"ID":"4532e5bf-184a-4cb3-b583-8451ff8a014e","Type":"ContainerStarted","Data":"e3a8a06f0b8ad5ce1290e906a60aa59830baff21b99ba62cfa11c6f9a1366bbb"} Jul 20 22:53:53.101018 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.100992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6lr4" event={"ID":"dbba1515-089c-438d-8ba9-56dd9d35f6b1","Type":"ContainerStarted","Data":"4832c793679fc62290dd6d55f015a3f02e6f0d74e13b7c4c1f2e0118f38ed533"} Jul 20 22:53:53.112476 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.112432 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.ec2.internal" podStartSLOduration=21.112418416 podStartE2EDuration="21.112418416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:53.112306046 +0000 UTC m=+22.608760470" watchObservedRunningTime="2026-07-20 22:53:53.112418416 +0000 UTC m=+22.608872837" Jul 20 22:53:53.144017 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.143977 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b6lr4" podStartSLOduration=1.6798055619999999 podStartE2EDuration="22.143962642s" totalImagesPullingTime="20.46415708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:53.128695775 +0000 UTC m=+22.625150195" watchObservedRunningTime="2026-07-20 22:53:53.143962642 +0000 UTC m=+22.640417084" Jul 20 22:53:53.144268 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:53.144249 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tsrlf" podStartSLOduration=2.044812534 podStartE2EDuration="22.144244487s" totalImagesPullingTime="20.099431953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:53.144197534 +0000 UTC m=+22.640651958" watchObservedRunningTime="2026-07-20 22:53:53.144244487 +0000 UTC m=+22.640698911" Jul 20 22:53:54.105379 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.105025 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjsrr" event={"ID":"feb100de-3890-47c2-868d-79a783d51b47","Type":"ContainerStarted","Data":"fe1a7a1ebbdb024a13a53f63932ecf737a79426b80a1ba3a14029c1f53c504be"} Jul 20 22:53:54.106703 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.106663 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c326231-6e02-4973-aa69-116110666151" containerID="8fd32a5a10c4224c0f42ae44158a16ec57a2a1328061c2a873449d3ccdc991b3" exitCode=0 Jul 20 22:53:54.106820 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.106742 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerDied","Data":"8fd32a5a10c4224c0f42ae44158a16ec57a2a1328061c2a873449d3ccdc991b3"} Jul 20 22:53:54.108290 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.108261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vwqjp" event={"ID":"6c0a5da1-61a9-4426-9646-800e40d3482c","Type":"ContainerStarted","Data":"2ba01ef500eb5fea59a04f3bc868a8b5cbead7986e1fb80a1bc61ebf4f5e536f"} Jul 20 22:53:54.111185 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111164 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 22:53:54.111530 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111512 2574 generic.go:356] "Generic (PLEG): container finished" podID="6caa92d9-60c0-4340-8426-2f60704881bf" containerID="1b34fd4ce28145df5d9781c8ebb399a8cb061b893857da52ca23096af2490b67" exitCode=1 Jul 20 22:53:54.111595 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"2474ed91b09288b2001967894521e3374731bead6541685f22e68f51d9d06d8a"} Jul 20 22:53:54.111651 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"039cc5d0c1445d5beb52e53440fd1c8c4c323f3c8f257b4b85f479c39c607879"} Jul 20 22:53:54.111696 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"b0960f09643f8a8992b5ef7329c769f855260206ac1e178dfe68f5c635c1935a"} Jul 20 22:53:54.111732 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"d2f6e2e482cdf7fba5cd1b729f8a74ce378fcb148ca9df580eb7abe7062915e0"} Jul 20 22:53:54.111780 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerDied","Data":"1b34fd4ce28145df5d9781c8ebb399a8cb061b893857da52ca23096af2490b67"} Jul 20 22:53:54.111780 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.111776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"d5c8e5298f8a17402edad398b3b4a6a54384e267c6395cb75738233a87cbb0fc"} Jul 20 22:53:54.113047 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.113027 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nnwwg" event={"ID":"4d5f7832-f567-42b5-966f-4084d961c60b","Type":"ContainerStarted","Data":"1aa28c75fd2162c545dcf4dd7a7695e57d6feaa66c41ed9032b3a89425dd6372"} Jul 20 22:53:54.114617 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.114596 2574 generic.go:356] "Generic (PLEG): container finished" podID="09139888c4ca3c1b45f388babb2f2bd5" containerID="55eff6727d56e069f7d973c534d953692ec8ad9da520cbde0d9b77a1eab4560e" exitCode=0 Jul 20 22:53:54.114718 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.114675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" event={"ID":"09139888c4ca3c1b45f388babb2f2bd5","Type":"ContainerDied","Data":"55eff6727d56e069f7d973c534d953692ec8ad9da520cbde0d9b77a1eab4560e"} Jul 20 22:53:54.114781 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.114772 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" Jul 20 22:53:54.116001 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.115982 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" event={"ID":"b1678de8-3d36-424d-b897-81373118ab57","Type":"ContainerStarted","Data":"6da95879a7d8c2401898976943761b7949fcd0080511eafb5a4adec4b849d4eb"} Jul 20 22:53:54.124755 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.124740 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 22:53:54.125476 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.125436 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zjsrr" podStartSLOduration=3.157919467 podStartE2EDuration="23.12542256s" totalImagesPullingTime="19.967503093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:54.119164408 +0000 UTC m=+23.615618833" watchObservedRunningTime="2026-07-20 22:53:54.12542256 +0000 UTC m=+23.621876983" Jul 20 22:53:54.125692 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.125678 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal"] Jul 20 22:53:54.133119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.132955 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vwqjp" podStartSLOduration=3.026392032 podStartE2EDuration="23.132941861s" totalImagesPullingTime="20.106549829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:54.132605115 +0000 UTC m=+23.629059543" watchObservedRunningTime="2026-07-20 22:53:54.132941861 +0000 UTC m=+23.629396331" Jul 20 22:53:54.147880 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.147840 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nnwwg" podStartSLOduration=11.021569169 podStartE2EDuration="23.147822369s" totalImagesPullingTime="12.1262532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:54.14776609 +0000 UTC m=+23.644220516" watchObservedRunningTime="2026-07-20 22:53:54.147822369 +0000 UTC m=+23.644276788" Jul 20 22:53:54.985074 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:54.985039 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 22:53:55.002928 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.002881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:55.003082 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.002890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:55.003082 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:55.003008 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:55.003082 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.003039 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:55.003242 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:55.003119 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:55.003242 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:55.003193 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:55.119869 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.119782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghqts" event={"ID":"ac8deb1c-423a-4303-95a7-f9a8fb23a892","Type":"ContainerStarted","Data":"ee4bc1fec7ae34356113d8f856f22cfc2d967b815c0873d37d40801e3ccad1c7"} Jul 20 22:53:55.121937 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.121903 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" event={"ID":"09139888c4ca3c1b45f388babb2f2bd5","Type":"ContainerStarted","Data":"c8f666c0c9b8f98db1fdce9c8523aed5b82c9dcef472ab6cf6b0b3a96e098bc6"} Jul 20 22:53:55.124030 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.123996 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" event={"ID":"b1678de8-3d36-424d-b897-81373118ab57","Type":"ContainerStarted","Data":"32c4846bde3e99650ba138bdfd34b3d668b9a51064343e6935c354d0ab7822e3"} Jul 20 22:53:55.147317 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.147261 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ghqts" podStartSLOduration=3.999289847 podStartE2EDuration="24.147217221s" totalImagesPullingTime="20.147927374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:55.14646899 +0000 UTC m=+24.642923414" watchObservedRunningTime="2026-07-20 22:53:55.147217221 +0000 UTC m=+24.643671645" Jul 20 22:53:55.160055 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.160001 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.ec2.internal" podStartSLOduration=1.159985071 podStartE2EDuration="1.159985071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:55.159317922 +0000 UTC m=+24.655772346" watchObservedRunningTime="2026-07-20 22:53:55.159985071 +0000 UTC m=+24.656439496" Jul 20 22:53:55.329184 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.329157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:55.329800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.329784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:55.973399 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.973273 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T22:53:54.985061991Z","UUID":"ae34b802-294c-408c-ab7f-32549c3944aa","Handler":null,"Name":"","Endpoint":""} Jul 20 22:53:55.974944 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.974917 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 22:53:55.975073 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:55.974955 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 22:53:56.125944 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:56.125910 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:56.126711 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:56.126694 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nnwwg" Jul 20 22:53:57.003897 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:57.003658 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:57.004182 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:57.003716 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:57.004182 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:57.003993 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:57.004182 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:57.003725 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:57.004182 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:57.004068 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:57.004182 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:57.004171 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:53:57.134065 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:57.134035 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 22:53:57.134481 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:57.134449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"4ab56e1fb19ae225a27b24094604bbf604ead3afa6bd028ee6a348314402d238"} Jul 20 22:53:57.136299 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:57.136268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" event={"ID":"b1678de8-3d36-424d-b897-81373118ab57","Type":"ContainerStarted","Data":"8954460b11361a9d1ff1c720da28d76f805ec4d9bd3736e6351b8cb26e3cbf2c"} Jul 20 22:53:59.003810 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:59.003777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:53:59.004320 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:59.003777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:53:59.004320 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:59.003909 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:53:59.004320 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:53:59.003926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:53:59.004320 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:59.004041 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:53:59.004320 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:53:59.004157 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:54:00.146616 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.146417 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c326231-6e02-4973-aa69-116110666151" containerID="beebb6a9b5af1c226795feace6c4dfee6561c9a0fd56e2fba17aa6dfebfdf3b3" exitCode=0 Jul 20 22:54:00.147336 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.146516 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerDied","Data":"beebb6a9b5af1c226795feace6c4dfee6561c9a0fd56e2fba17aa6dfebfdf3b3"} Jul 20 22:54:00.150119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.150104 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 22:54:00.150446 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.150430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"a5fe10eed5fb73d12a0e9ea284148277fefa22d3281cb5c7c4a3f5d31f1b8f98"} Jul 20 22:54:00.150664 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.150650 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:54:00.150725 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.150715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:54:00.150855 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.150835 2574 scope.go:117] "RemoveContainer" containerID="1b34fd4ce28145df5d9781c8ebb399a8cb061b893857da52ca23096af2490b67" Jul 20 22:54:00.166164 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.166145 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:54:00.168994 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:00.168955 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt55b" podStartSLOduration=6.066094965 podStartE2EDuration="29.168938755s" totalImagesPullingTime="23.10284379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:53:57.153389801 +0000 UTC m=+26.649844248" watchObservedRunningTime="2026-07-20 22:54:00.168938755 +0000 UTC m=+29.665393179" Jul 20 22:54:01.004384 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.003935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:01.004384 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.004023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:01.004384 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:01.004039 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:54:01.004384 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:01.004115 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:54:01.004384 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.004144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:01.004384 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:01.004214 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:54:01.156923 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.156899 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 22:54:01.157371 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.157244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" event={"ID":"6caa92d9-60c0-4340-8426-2f60704881bf","Type":"ContainerStarted","Data":"baf04ac69272f6a970f4d1e93834fe8c1974cad6b3ddbff5f50f38227afe359e"} Jul 20 22:54:01.157524 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.157483 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:54:01.172653 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.172631 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:54:01.186941 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.186901 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" podStartSLOduration=9.318683318 podStartE2EDuration="30.186888044s" totalImagesPullingTime="20.868204726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:01.186534338 +0000 UTC m=+30.682988761" watchObservedRunningTime="2026-07-20 22:54:01.186888044 +0000 UTC m=+30.683342467" Jul 20 22:54:01.552180 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.552109 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xl88f"] Jul 20 22:54:01.552299 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.552231 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:01.552336 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:01.552306 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:54:01.555303 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.555277 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-88x7j"] Jul 20 22:54:01.555439 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.555371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:01.555534 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:01.555457 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:54:01.558105 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.558082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gw7vz"] Jul 20 22:54:01.558206 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:01.558165 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:01.558255 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:01.558238 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:54:02.161363 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:02.161328 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c326231-6e02-4973-aa69-116110666151" containerID="0cedec526e06a13f7f67c5ffcfb6cce2dc17ab69e78de37829fe9a6ef20aeff0" exitCode=0 Jul 20 22:54:02.161743 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:02.161407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerDied","Data":"0cedec526e06a13f7f67c5ffcfb6cce2dc17ab69e78de37829fe9a6ef20aeff0"} Jul 20 22:54:03.003339 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:03.003128 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:03.003470 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:03.003134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:03.003470 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.003382 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:54:03.003470 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:03.003134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:03.003470 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.003451 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:54:03.003656 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.003540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:54:03.605422 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:03.605392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:03.605909 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:03.605446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:03.605909 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.605558 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:54:03.605909 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.605571 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 22:54:03.605909 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.605609 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs podName:2d4e33a0-0442-47b0-9ecf-e833f909a185 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:35.605595965 +0000 UTC m=+65.102050367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs") pod "network-metrics-daemon-gw7vz" (UID: "2d4e33a0-0442-47b0-9ecf-e833f909a185") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 22:54:03.605909 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.605634 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret podName:ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:35.605617231 +0000 UTC m=+65.102071633 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret") pod "global-pull-secret-syncer-xl88f" (UID: "ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39") : object "kube-system"/"original-pull-secret" not registered Jul 20 22:54:03.705896 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:03.705867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:03.706044 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.705979 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 22:54:03.706044 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.705993 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 22:54:03.706044 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.706002 2574 projected.go:196] Error preparing data for projected volume kube-api-access-wxv8w for pod openshift-network-diagnostics/network-check-target-88x7j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:54:03.706157 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:03.706047 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w podName:fcc384bb-5ccb-4609-948b-88c41d22bb2a nodeName:}" failed. No retries permitted until 2026-07-20 22:54:35.706033967 +0000 UTC m=+65.202488369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxv8w" (UniqueName: "kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w") pod "network-check-target-88x7j" (UID: "fcc384bb-5ccb-4609-948b-88c41d22bb2a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 22:54:04.168629 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:04.168592 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c326231-6e02-4973-aa69-116110666151" containerID="5a1aa46d2a925b3c8fee3032b8c23dcd0f19273eaffa67475fc647376430848f" exitCode=0 Jul 20 22:54:04.168804 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:04.168647 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerDied","Data":"5a1aa46d2a925b3c8fee3032b8c23dcd0f19273eaffa67475fc647376430848f"} Jul 20 22:54:05.003657 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.003620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:05.004082 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.003666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:05.004082 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.003620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:05.004082 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:05.003773 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gw7vz" podUID="2d4e33a0-0442-47b0-9ecf-e833f909a185" Jul 20 22:54:05.004082 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:05.003901 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-88x7j" podUID="fcc384bb-5ccb-4609-948b-88c41d22bb2a" Jul 20 22:54:05.004082 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:05.003982 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xl88f" podUID="ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39" Jul 20 22:54:05.871699 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.871670 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-37.ec2.internal" event="NodeReady" Jul 20 22:54:05.871867 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.871825 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 22:54:05.917941 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.917531 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b46cc44c9-brbhm"] Jul 20 22:54:05.921099 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.921067 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf"] Jul 20 22:54:05.921213 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.921184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:05.923673 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.923652 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w7p55\"" Jul 20 22:54:05.923774 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.923746 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 22:54:05.924077 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.924035 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 22:54:05.924183 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.924095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 22:54:05.925346 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.925322 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-ncfsk"] Jul 20 22:54:05.925469 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.925453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" Jul 20 22:54:05.927569 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.927552 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:54:05.927685 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.927663 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 20 22:54:05.927685 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.927664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8dk7r\"" Jul 20 22:54:05.930918 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.929588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:05.930918 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.929835 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 22:54:05.931929 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.931911 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 20 22:54:05.932445 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.932302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 20 22:54:05.932445 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.932310 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7x4c6\"" Jul 20 22:54:05.932445 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.932351 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 20 22:54:05.932445 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.932361 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:54:05.939781 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.939707 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25"] Jul 20 22:54:05.942380 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.942357 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 20 22:54:05.945587 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.945563 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m"] Jul 20 22:54:05.945796 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.945777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:05.949453 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.949435 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx"] Jul 20 22:54:05.950528 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.949719 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 20 22:54:05.950528 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.949837 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:54:05.950528 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.949837 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-97j7x\"" Jul 20 22:54:05.950528 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.949958 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 20 22:54:05.950528 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.950061 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:05.950528 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.950234 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 20 22:54:05.952397 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.952377 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 20 22:54:05.952732 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.952714 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 20 22:54:05.954785 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.954766 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4"] Jul 20 22:54:05.955110 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.955091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" Jul 20 22:54:05.955598 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.955575 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:54:05.956550 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.955984 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sw6kr\"" Jul 20 22:54:05.956550 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.956194 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 20 22:54:05.957655 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.957626 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 22:54:05.959174 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.958408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf"] Jul 20 22:54:05.959174 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.958443 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b46cc44c9-brbhm"] Jul 20 22:54:05.959174 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.958464 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx"] Jul 20 22:54:05.959174 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.958485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25"] Jul 20 22:54:05.959174 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.958561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:05.959505 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.959464 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-98frh\"" Jul 20 22:54:05.959608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.959553 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 22:54:05.961516 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.961476 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 20 22:54:05.961798 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.961552 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fzdrm\"" Jul 20 22:54:05.962319 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.962303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 20 22:54:05.963976 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.963959 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jlh2g"] Jul 20 22:54:05.967861 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.967842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:05.969930 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.969909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4"] Jul 20 22:54:05.970816 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.970793 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-ncfsk"] Jul 20 22:54:05.971010 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.970991 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qkglg\"" Jul 20 22:54:05.971077 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.971010 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 22:54:05.971077 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.971047 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 22:54:05.971255 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.971238 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 22:54:05.976636 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.976616 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m"] Jul 20 22:54:05.980591 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:05.980570 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jlh2g"] Jul 20 22:54:06.024748 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.024718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqwf\" (UniqueName: \"kubernetes.io/projected/3eed9716-0a22-48ec-aab4-850e2656a381-kube-api-access-cxqwf\") pod \"volume-data-source-validator-78cf4c6f6-s84pf\" (UID: \"3eed9716-0a22-48ec-aab4-850e2656a381\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" Jul 20 22:54:06.025275 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.024762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/077d3a13-d78c-4672-84c2-7dd2a551f322-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.025275 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.024876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-installation-pull-secrets\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025414 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-bound-sa-token\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025548 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-certificates\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025595 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025641 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psm8b\" (UniqueName: \"kubernetes.io/projected/077d3a13-d78c-4672-84c2-7dd2a551f322-kube-api-access-psm8b\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.025684 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c79080fd-58d2-4bec-a34c-476ef421391d-config\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.025733 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/077d3a13-d78c-4672-84c2-7dd2a551f322-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.025733 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9kz\" (UniqueName: \"kubernetes.io/projected/c79080fd-58d2-4bec-a34c-476ef421391d-kube-api-access-vl9kz\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.025810 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmp4\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-kube-api-access-zvmp4\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025810 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41a5f424-012a-41c7-ad86-424296e7b06d-trusted-ca\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.025884 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-image-registry-private-configuration\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025884 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41a5f424-012a-41c7-ad86-424296e7b06d-config\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.025884 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41a5f424-012a-41c7-ad86-424296e7b06d-serving-cert\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.025985 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vd2\" (UniqueName: \"kubernetes.io/projected/41a5f424-012a-41c7-ad86-424296e7b06d-kube-api-access-w4vd2\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.025985 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb3a9d52-6223-4a61-94c0-066567aa187f-ca-trust-extracted\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025985 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-trusted-ca\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.025985 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.025979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l2cn\" (UniqueName: \"kubernetes.io/projected/923af8cd-873d-43fe-8764-da0fe285838e-kube-api-access-4l2cn\") pod \"network-check-source-7bc5b6bf85-2kfdx\" (UID: \"923af8cd-873d-43fe-8764-da0fe285838e\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" Jul 20 22:54:06.026115 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.026006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c79080fd-58d2-4bec-a34c-476ef421391d-serving-cert\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.034797 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.034772 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lzh55"] Jul 20 22:54:06.038557 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.038524 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.040972 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.040821 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 22:54:06.040972 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.040860 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 22:54:06.040972 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.040881 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtmm9\"" Jul 20 22:54:06.048625 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.048604 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lzh55"] Jul 20 22:54:06.126545 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:06.126741 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv8lx\" (UniqueName: \"kubernetes.io/projected/cba5b0a1-8585-46be-9dad-bc766849f281-kube-api-access-jv8lx\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.126741 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-image-registry-private-configuration\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.126741 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cba5b0a1-8585-46be-9dad-bc766849f281-config-volume\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.126741 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41a5f424-012a-41c7-ad86-424296e7b06d-config\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.126741 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba5b0a1-8585-46be-9dad-bc766849f281-tmp-dir\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vd2\" (UniqueName: \"kubernetes.io/projected/41a5f424-012a-41c7-ad86-424296e7b06d-kube-api-access-w4vd2\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c79080fd-58d2-4bec-a34c-476ef421391d-serving-cert\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb3a9d52-6223-4a61-94c0-066567aa187f-ca-trust-extracted\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l2cn\" (UniqueName: \"kubernetes.io/projected/923af8cd-873d-43fe-8764-da0fe285838e-kube-api-access-4l2cn\") pod \"network-check-source-7bc5b6bf85-2kfdx\" (UID: \"923af8cd-873d-43fe-8764-da0fe285838e\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efb0d356-51ab-4dd1-b549-084d6c826bd8-cert\") pod \"ingress-canary-jlh2g\" (UID: \"efb0d356-51ab-4dd1-b549-084d6c826bd8\") " pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psm8b\" (UniqueName: \"kubernetes.io/projected/077d3a13-d78c-4672-84c2-7dd2a551f322-kube-api-access-psm8b\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c79080fd-58d2-4bec-a34c-476ef421391d-config\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9kz\" (UniqueName: \"kubernetes.io/projected/c79080fd-58d2-4bec-a34c-476ef421391d-kube-api-access-vl9kz\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.127004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.126991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmp4\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-kube-api-access-zvmp4\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41a5f424-012a-41c7-ad86-424296e7b06d-trusted-ca\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41a5f424-012a-41c7-ad86-424296e7b06d-serving-cert\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-trusted-ca\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqwf\" (UniqueName: \"kubernetes.io/projected/3eed9716-0a22-48ec-aab4-850e2656a381-kube-api-access-cxqwf\") pod \"volume-data-source-validator-78cf4c6f6-s84pf\" (UID: \"3eed9716-0a22-48ec-aab4-850e2656a381\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/077d3a13-d78c-4672-84c2-7dd2a551f322-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-installation-pull-secrets\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-bound-sa-token\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-certificates\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.127398 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/077d3a13-d78c-4672-84c2-7dd2a551f322-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.127791 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41a5f424-012a-41c7-ad86-424296e7b06d-config\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.128113 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jls5d\" (UniqueName: \"kubernetes.io/projected/efb0d356-51ab-4dd1-b549-084d6c826bd8-kube-api-access-jls5d\") pod \"ingress-canary-jlh2g\" (UID: \"efb0d356-51ab-4dd1-b549-084d6c826bd8\") " pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:06.128113 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.127983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.128113 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.128006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.128314 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.128291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/077d3a13-d78c-4672-84c2-7dd2a551f322-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.128375 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.128319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c79080fd-58d2-4bec-a34c-476ef421391d-config\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.128441 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.128420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41a5f424-012a-41c7-ad86-424296e7b06d-trusted-ca\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.128637 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.128616 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb3a9d52-6223-4a61-94c0-066567aa187f-ca-trust-extracted\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.128754 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.128689 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 22:54:06.128754 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.128709 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b46cc44c9-brbhm: secret "image-registry-tls" not found Jul 20 22:54:06.128853 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.128766 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls podName:bb3a9d52-6223-4a61-94c0-066567aa187f nodeName:}" failed. No retries permitted until 2026-07-20 22:54:06.628748012 +0000 UTC m=+36.125202416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls") pod "image-registry-b46cc44c9-brbhm" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f") : secret "image-registry-tls" not found Jul 20 22:54:06.129119 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.129094 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-certificates\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.129726 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.129652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-trusted-ca\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.131651 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.131439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c79080fd-58d2-4bec-a34c-476ef421391d-serving-cert\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.131651 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.131526 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41a5f424-012a-41c7-ad86-424296e7b06d-serving-cert\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.131814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.131704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-installation-pull-secrets\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.131814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.131712 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-image-registry-private-configuration\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.131814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.131764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/077d3a13-d78c-4672-84c2-7dd2a551f322-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.137111 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.137070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmp4\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-kube-api-access-zvmp4\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.137510 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.137468 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vd2\" (UniqueName: \"kubernetes.io/projected/41a5f424-012a-41c7-ad86-424296e7b06d-kube-api-access-w4vd2\") pod \"console-operator-565b7b569d-ncfsk\" (UID: \"41a5f424-012a-41c7-ad86-424296e7b06d\") " pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.138185 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.138160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psm8b\" (UniqueName: \"kubernetes.io/projected/077d3a13-d78c-4672-84c2-7dd2a551f322-kube-api-access-psm8b\") pod \"kube-storage-version-migrator-operator-7658d9b749-84s25\" (UID: \"077d3a13-d78c-4672-84c2-7dd2a551f322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.138611 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.138580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-bound-sa-token\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.139309 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.139270 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9kz\" (UniqueName: \"kubernetes.io/projected/c79080fd-58d2-4bec-a34c-476ef421391d-kube-api-access-vl9kz\") pod \"service-ca-operator-5bcbc94c78-6rh4m\" (UID: \"c79080fd-58d2-4bec-a34c-476ef421391d\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.140129 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.139638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l2cn\" (UniqueName: \"kubernetes.io/projected/923af8cd-873d-43fe-8764-da0fe285838e-kube-api-access-4l2cn\") pod \"network-check-source-7bc5b6bf85-2kfdx\" (UID: \"923af8cd-873d-43fe-8764-da0fe285838e\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" Jul 20 22:54:06.140129 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.139919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqwf\" (UniqueName: \"kubernetes.io/projected/3eed9716-0a22-48ec-aab4-850e2656a381-kube-api-access-cxqwf\") pod \"volume-data-source-validator-78cf4c6f6-s84pf\" (UID: \"3eed9716-0a22-48ec-aab4-850e2656a381\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" Jul 20 22:54:06.168140 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.168055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nnwwg_4d5f7832-f567-42b5-966f-4084d961c60b/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:06.229062 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:06.229224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jls5d\" (UniqueName: \"kubernetes.io/projected/efb0d356-51ab-4dd1-b549-084d6c826bd8-kube-api-access-jls5d\") pod \"ingress-canary-jlh2g\" (UID: \"efb0d356-51ab-4dd1-b549-084d6c826bd8\") " pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:06.229224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.229224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:06.229224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv8lx\" (UniqueName: \"kubernetes.io/projected/cba5b0a1-8585-46be-9dad-bc766849f281-kube-api-access-jv8lx\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.229224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cba5b0a1-8585-46be-9dad-bc766849f281-config-volume\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.229224 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba5b0a1-8585-46be-9dad-bc766849f281-tmp-dir\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.229532 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efb0d356-51ab-4dd1-b549-084d6c826bd8-cert\") pod \"ingress-canary-jlh2g\" (UID: \"efb0d356-51ab-4dd1-b549-084d6c826bd8\") " pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:06.229532 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.229271 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 22:54:06.229532 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.229348 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 22:54:06.229532 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.229350 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert podName:2e4c6d2b-cce8-4b70-af8d-8791605ee28e nodeName:}" failed. No retries permitted until 2026-07-20 22:54:06.7293303 +0000 UTC m=+36.225784719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tgjs4" (UID: "2e4c6d2b-cce8-4b70-af8d-8791605ee28e") : secret "networking-console-plugin-cert" not found Jul 20 22:54:06.229532 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.229535 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls podName:cba5b0a1-8585-46be-9dad-bc766849f281 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:06.729517312 +0000 UTC m=+36.225971714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls") pod "dns-default-lzh55" (UID: "cba5b0a1-8585-46be-9dad-bc766849f281") : secret "dns-default-metrics-tls" not found Jul 20 22:54:06.229795 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229698 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba5b0a1-8585-46be-9dad-bc766849f281-tmp-dir\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.229845 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:06.229933 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.229912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cba5b0a1-8585-46be-9dad-bc766849f281-config-volume\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.231686 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.231670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efb0d356-51ab-4dd1-b549-084d6c826bd8-cert\") pod \"ingress-canary-jlh2g\" (UID: \"efb0d356-51ab-4dd1-b549-084d6c826bd8\") " pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:06.238876 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.238854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jls5d\" (UniqueName: \"kubernetes.io/projected/efb0d356-51ab-4dd1-b549-084d6c826bd8-kube-api-access-jls5d\") pod \"ingress-canary-jlh2g\" (UID: \"efb0d356-51ab-4dd1-b549-084d6c826bd8\") " pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:06.240237 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.240220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv8lx\" (UniqueName: \"kubernetes.io/projected/cba5b0a1-8585-46be-9dad-bc766849f281-kube-api-access-jv8lx\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.248166 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.248145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" Jul 20 22:54:06.255450 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.255433 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:06.256116 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.256069 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nnwwg_4d5f7832-f567-42b5-966f-4084d961c60b/konnectivity-agent/0.log" limit=1024 Jul 20 22:54:06.264768 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.264747 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" Jul 20 22:54:06.274835 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.274816 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" Jul 20 22:54:06.282257 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.282105 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" Jul 20 22:54:06.295308 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.295272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jlh2g" Jul 20 22:54:06.477717 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.477225 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf"] Jul 20 22:54:06.478257 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.478220 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-ncfsk"] Jul 20 22:54:06.525046 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.525013 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jlh2g"] Jul 20 22:54:06.528073 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:06.528045 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb0d356_51ab_4dd1_b549_084d6c826bd8.slice/crio-a5d5b30cdeffab94e3a7028fdaab6f37ad676a50156b199925365339289035f8 WatchSource:0}: Error finding container a5d5b30cdeffab94e3a7028fdaab6f37ad676a50156b199925365339289035f8: Status 404 returned error can't find the container with id a5d5b30cdeffab94e3a7028fdaab6f37ad676a50156b199925365339289035f8 Jul 20 22:54:06.531527 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.531482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx"] Jul 20 22:54:06.532775 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.532752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25"] Jul 20 22:54:06.535503 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.535467 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m"] Jul 20 22:54:06.536329 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:06.536297 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923af8cd_873d_43fe_8764_da0fe285838e.slice/crio-3a0c932121edbacd624f5938b985a90ae7907d3013d7a68c9b88eca0e2463d9c WatchSource:0}: Error finding container 3a0c932121edbacd624f5938b985a90ae7907d3013d7a68c9b88eca0e2463d9c: Status 404 returned error can't find the container with id 3a0c932121edbacd624f5938b985a90ae7907d3013d7a68c9b88eca0e2463d9c Jul 20 22:54:06.537398 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:06.537373 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077d3a13_d78c_4672_84c2_7dd2a551f322.slice/crio-02683f943dc00749b1ad22eba729c892527560277b8625ce5774e5bce4c3d517 WatchSource:0}: Error finding container 02683f943dc00749b1ad22eba729c892527560277b8625ce5774e5bce4c3d517: Status 404 returned error can't find the container with id 02683f943dc00749b1ad22eba729c892527560277b8625ce5774e5bce4c3d517 Jul 20 22:54:06.538044 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:06.538022 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79080fd_58d2_4bec_a34c_476ef421391d.slice/crio-3ca0093475623c1e81ade6419c4d3f600a948b318c95b58500263d01a7144916 WatchSource:0}: Error finding container 3ca0093475623c1e81ade6419c4d3f600a948b318c95b58500263d01a7144916: Status 404 returned error can't find the container with id 3ca0093475623c1e81ade6419c4d3f600a948b318c95b58500263d01a7144916 Jul 20 22:54:06.633650 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.633616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:06.633804 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.633731 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 22:54:06.633804 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.633743 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b46cc44c9-brbhm: secret "image-registry-tls" not found Jul 20 22:54:06.633804 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.633788 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls podName:bb3a9d52-6223-4a61-94c0-066567aa187f nodeName:}" failed. No retries permitted until 2026-07-20 22:54:07.63377607 +0000 UTC m=+37.130230472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls") pod "image-registry-b46cc44c9-brbhm" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f") : secret "image-registry-tls" not found Jul 20 22:54:06.734971 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.734892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:06.734971 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:06.734930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:06.735143 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.735047 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 22:54:06.735143 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.735076 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 22:54:06.735143 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.735112 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls podName:cba5b0a1-8585-46be-9dad-bc766849f281 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:07.735095383 +0000 UTC m=+37.231549793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls") pod "dns-default-lzh55" (UID: "cba5b0a1-8585-46be-9dad-bc766849f281") : secret "dns-default-metrics-tls" not found Jul 20 22:54:06.735143 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:06.735127 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert podName:2e4c6d2b-cce8-4b70-af8d-8791605ee28e nodeName:}" failed. No retries permitted until 2026-07-20 22:54:07.735120778 +0000 UTC m=+37.231575180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tgjs4" (UID: "2e4c6d2b-cce8-4b70-af8d-8791605ee28e") : secret "networking-console-plugin-cert" not found Jul 20 22:54:07.007627 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.007138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:07.007627 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.007194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:07.007627 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.007160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:07.009790 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.009765 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d688j\"" Jul 20 22:54:07.009900 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.009828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 22:54:07.010066 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.010049 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xr56f\"" Jul 20 22:54:07.010264 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.010240 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 22:54:07.183154 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.183104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" event={"ID":"c79080fd-58d2-4bec-a34c-476ef421391d","Type":"ContainerStarted","Data":"3ca0093475623c1e81ade6419c4d3f600a948b318c95b58500263d01a7144916"} Jul 20 22:54:07.187355 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.187326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" event={"ID":"923af8cd-873d-43fe-8764-da0fe285838e","Type":"ContainerStarted","Data":"3a0c932121edbacd624f5938b985a90ae7907d3013d7a68c9b88eca0e2463d9c"} Jul 20 22:54:07.189217 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.189194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" event={"ID":"077d3a13-d78c-4672-84c2-7dd2a551f322","Type":"ContainerStarted","Data":"02683f943dc00749b1ad22eba729c892527560277b8625ce5774e5bce4c3d517"} Jul 20 22:54:07.191963 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.191896 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" event={"ID":"41a5f424-012a-41c7-ad86-424296e7b06d","Type":"ContainerStarted","Data":"5848c889886d40b9663c26d6197fc9749d1ef4f2be9ec33e9a7fa0fe9083a1f6"} Jul 20 22:54:07.194231 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.194171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jlh2g" event={"ID":"efb0d356-51ab-4dd1-b549-084d6c826bd8","Type":"ContainerStarted","Data":"a5d5b30cdeffab94e3a7028fdaab6f37ad676a50156b199925365339289035f8"} Jul 20 22:54:07.196435 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.196371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" event={"ID":"3eed9716-0a22-48ec-aab4-850e2656a381","Type":"ContainerStarted","Data":"c9cf624a18ee3c0bf0cdf0810c1fb26678089f966ff67926c28c003c15d9e26c"} Jul 20 22:54:07.645202 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.645144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:07.651511 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:07.649625 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 22:54:07.651511 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:07.649650 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b46cc44c9-brbhm: secret "image-registry-tls" not found Jul 20 22:54:07.651511 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:07.649722 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls podName:bb3a9d52-6223-4a61-94c0-066567aa187f nodeName:}" failed. No retries permitted until 2026-07-20 22:54:09.649701389 +0000 UTC m=+39.146155810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls") pod "image-registry-b46cc44c9-brbhm" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f") : secret "image-registry-tls" not found Jul 20 22:54:07.747441 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.746633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:07.747441 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:07.746694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:07.747441 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:07.746843 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 22:54:07.747441 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:07.746909 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert podName:2e4c6d2b-cce8-4b70-af8d-8791605ee28e nodeName:}" failed. No retries permitted until 2026-07-20 22:54:09.746889269 +0000 UTC m=+39.243343694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tgjs4" (UID: "2e4c6d2b-cce8-4b70-af8d-8791605ee28e") : secret "networking-console-plugin-cert" not found Jul 20 22:54:07.747441 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:07.747333 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 22:54:07.747441 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:07.747381 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls podName:cba5b0a1-8585-46be-9dad-bc766849f281 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:09.747367129 +0000 UTC m=+39.243821536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls") pod "dns-default-lzh55" (UID: "cba5b0a1-8585-46be-9dad-bc766849f281") : secret "dns-default-metrics-tls" not found Jul 20 22:54:09.666771 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:09.666731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:09.667282 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:09.666910 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 22:54:09.667282 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:09.666933 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b46cc44c9-brbhm: secret "image-registry-tls" not found Jul 20 22:54:09.667282 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:09.667003 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls podName:bb3a9d52-6223-4a61-94c0-066567aa187f nodeName:}" failed. No retries permitted until 2026-07-20 22:54:13.666982313 +0000 UTC m=+43.163436719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls") pod "image-registry-b46cc44c9-brbhm" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f") : secret "image-registry-tls" not found Jul 20 22:54:09.767634 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:09.767595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:09.767634 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:09.767639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:09.767881 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:09.767722 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 22:54:09.767881 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:09.767776 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls podName:cba5b0a1-8585-46be-9dad-bc766849f281 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:13.767763292 +0000 UTC m=+43.264217698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls") pod "dns-default-lzh55" (UID: "cba5b0a1-8585-46be-9dad-bc766849f281") : secret "dns-default-metrics-tls" not found Jul 20 22:54:09.767881 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:09.767722 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 22:54:09.767881 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:09.767857 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert podName:2e4c6d2b-cce8-4b70-af8d-8791605ee28e nodeName:}" failed. No retries permitted until 2026-07-20 22:54:13.767844969 +0000 UTC m=+43.264299371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tgjs4" (UID: "2e4c6d2b-cce8-4b70-af8d-8791605ee28e") : secret "networking-console-plugin-cert" not found Jul 20 22:54:13.700578 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:13.700539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:13.701100 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:13.700751 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 22:54:13.701100 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:13.700770 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b46cc44c9-brbhm: secret "image-registry-tls" not found Jul 20 22:54:13.701100 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:13.700832 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls podName:bb3a9d52-6223-4a61-94c0-066567aa187f nodeName:}" failed. No retries permitted until 2026-07-20 22:54:21.70081239 +0000 UTC m=+51.197266792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls") pod "image-registry-b46cc44c9-brbhm" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f") : secret "image-registry-tls" not found Jul 20 22:54:13.800997 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:13.800956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:13.801196 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:13.801064 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 22:54:13.801196 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:13.801097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:13.801196 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:13.801129 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert podName:2e4c6d2b-cce8-4b70-af8d-8791605ee28e nodeName:}" failed. No retries permitted until 2026-07-20 22:54:21.801114989 +0000 UTC m=+51.297569395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tgjs4" (UID: "2e4c6d2b-cce8-4b70-af8d-8791605ee28e") : secret "networking-console-plugin-cert" not found Jul 20 22:54:13.801373 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:13.801238 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 22:54:13.801373 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:13.801314 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls podName:cba5b0a1-8585-46be-9dad-bc766849f281 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:21.801279942 +0000 UTC m=+51.297734355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls") pod "dns-default-lzh55" (UID: "cba5b0a1-8585-46be-9dad-bc766849f281") : secret "dns-default-metrics-tls" not found Jul 20 22:54:15.222206 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:15.222183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" event={"ID":"077d3a13-d78c-4672-84c2-7dd2a551f322","Type":"ContainerStarted","Data":"070ffbf3b83405fc0791169bc2a45d3786e9ba1cf2db58eac36b617379f4415e"} Jul 20 22:54:16.227213 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.227171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jlh2g" event={"ID":"efb0d356-51ab-4dd1-b549-084d6c826bd8","Type":"ContainerStarted","Data":"9b5704fb76f8666f325ccef00d2c615ed344c0f0d725d4fe5497ae482a88cc86"} Jul 20 22:54:16.228884 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.228851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" event={"ID":"3eed9716-0a22-48ec-aab4-850e2656a381","Type":"ContainerStarted","Data":"8d888a5f5c294b3195a46bd6a3e05e536ebba6fe11d96bb63370398ef3fa1576"} Jul 20 22:54:16.230440 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.230386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" event={"ID":"c79080fd-58d2-4bec-a34c-476ef421391d","Type":"ContainerStarted","Data":"a091f097521675c1a379dfeb86b3d54f2975872d252370fff57a82c7b3c85557"} Jul 20 22:54:16.233468 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.233442 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c326231-6e02-4973-aa69-116110666151" containerID="ca71979cf04da859a82aaeb578bc8e915bddd3f539c0eacb91d7b54321393d76" exitCode=0 Jul 20 22:54:16.233576 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.233527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerDied","Data":"ca71979cf04da859a82aaeb578bc8e915bddd3f539c0eacb91d7b54321393d76"} Jul 20 22:54:16.235257 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.235227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" event={"ID":"923af8cd-873d-43fe-8764-da0fe285838e","Type":"ContainerStarted","Data":"123c14fb6b1c6d1d639185f8e8a31679b708681d16ebc5c446f98d2c67175b07"} Jul 20 22:54:16.236920 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.236903 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/0.log" Jul 20 22:54:16.237004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.236935 2574 generic.go:356] "Generic (PLEG): container finished" podID="41a5f424-012a-41c7-ad86-424296e7b06d" containerID="ddeef0a930f4c556a52fa2359881c62f162f960c979be008766ed7686cabf3af" exitCode=255 Jul 20 22:54:16.237004 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.236995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" event={"ID":"41a5f424-012a-41c7-ad86-424296e7b06d","Type":"ContainerDied","Data":"ddeef0a930f4c556a52fa2359881c62f162f960c979be008766ed7686cabf3af"} Jul 20 22:54:16.237198 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.237184 2574 scope.go:117] "RemoveContainer" containerID="ddeef0a930f4c556a52fa2359881c62f162f960c979be008766ed7686cabf3af" Jul 20 22:54:16.242670 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.242634 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jlh2g" podStartSLOduration=2.701069884 podStartE2EDuration="11.242622223s" totalImagesPullingTime="8.541552339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:16.241084409 +0000 UTC m=+45.737538853" watchObservedRunningTime="2026-07-20 22:54:16.242622223 +0000 UTC m=+45.739076673" Jul 20 22:54:16.256102 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.256080 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:16.256210 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.256153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:16.259800 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.259597 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" podStartSLOduration=30.727611134 podStartE2EDuration="39.259581895s" totalImagesPullingTime="8.531970761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:16.258952312 +0000 UTC m=+45.755406760" watchObservedRunningTime="2026-07-20 22:54:16.259581895 +0000 UTC m=+45.756036317" Jul 20 22:54:16.275442 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.274997 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-s84pf" podStartSLOduration=35.690881084 podStartE2EDuration="44.274979556s" totalImagesPullingTime="8.584098472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:16.273147721 +0000 UTC m=+45.769602147" watchObservedRunningTime="2026-07-20 22:54:16.274979556 +0000 UTC m=+45.771433983" Jul 20 22:54:16.319991 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.319914 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" podStartSLOduration=29.787333191 podStartE2EDuration="38.319897748s" totalImagesPullingTime="8.532564557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:16.318643734 +0000 UTC m=+45.815098159" watchObservedRunningTime="2026-07-20 22:54:16.319897748 +0000 UTC m=+45.816352173" Jul 20 22:54:16.340817 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.340757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-2kfdx" podStartSLOduration=29.796430956000002 podStartE2EDuration="38.340735559s" totalImagesPullingTime="8.544304603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:16.33909353 +0000 UTC m=+45.835547959" watchObservedRunningTime="2026-07-20 22:54:16.340735559 +0000 UTC m=+45.837189984" Jul 20 22:54:16.767728 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.767640 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm"] Jul 20 22:54:16.790395 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.790352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm"] Jul 20 22:54:16.790596 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.790513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" Jul 20 22:54:16.792939 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.792913 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fk8z2\"" Jul 20 22:54:16.792939 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.792922 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 20 22:54:16.793134 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.793084 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 20 22:54:16.928241 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:16.928207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhth\" (UniqueName: \"kubernetes.io/projected/86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5-kube-api-access-pbhth\") pod \"migrator-7cc9c7ffd7-pdxhm\" (UID: \"86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" Jul 20 22:54:17.029262 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.029179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhth\" (UniqueName: \"kubernetes.io/projected/86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5-kube-api-access-pbhth\") pod \"migrator-7cc9c7ffd7-pdxhm\" (UID: \"86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" Jul 20 22:54:17.039918 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.039894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhth\" (UniqueName: \"kubernetes.io/projected/86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5-kube-api-access-pbhth\") pod \"migrator-7cc9c7ffd7-pdxhm\" (UID: \"86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" Jul 20 22:54:17.100912 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.100877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" Jul 20 22:54:17.244339 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.244310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm"] Jul 20 22:54:17.244733 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.244555 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c326231-6e02-4973-aa69-116110666151" containerID="dd03e9eda574a4a7a125e65f4b710d0570ca0b20cf67a692589f3dcc1a4bb0fc" exitCode=0 Jul 20 22:54:17.244733 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.244633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerDied","Data":"dd03e9eda574a4a7a125e65f4b710d0570ca0b20cf67a692589f3dcc1a4bb0fc"} Jul 20 22:54:17.246329 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.246308 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 22:54:17.246799 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.246778 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/0.log" Jul 20 22:54:17.246889 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.246820 2574 generic.go:356] "Generic (PLEG): container finished" podID="41a5f424-012a-41c7-ad86-424296e7b06d" containerID="0f9b6446ae51fea4f99d956292dcf8255f3f3b44fef45b5ac0ea25802c0081e1" exitCode=255 Jul 20 22:54:17.246958 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.246901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" event={"ID":"41a5f424-012a-41c7-ad86-424296e7b06d","Type":"ContainerDied","Data":"0f9b6446ae51fea4f99d956292dcf8255f3f3b44fef45b5ac0ea25802c0081e1"} Jul 20 22:54:17.247020 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.246996 2574 scope.go:117] "RemoveContainer" containerID="ddeef0a930f4c556a52fa2359881c62f162f960c979be008766ed7686cabf3af" Jul 20 22:54:17.247536 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:17.247154 2574 scope.go:117] "RemoveContainer" containerID="0f9b6446ae51fea4f99d956292dcf8255f3f3b44fef45b5ac0ea25802c0081e1" Jul 20 22:54:17.247536 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:17.247331 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-ncfsk_openshift-console-operator(41a5f424-012a-41c7-ad86-424296e7b06d)\"" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" podUID="41a5f424-012a-41c7-ad86-424296e7b06d" Jul 20 22:54:17.247724 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:17.247561 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d2e1d5_ec00_4b5a_9ed2_cefb3d4ac3b5.slice/crio-36ea6fefe0d077b06ba9f96ea3e7a816fa9d4a8c8e2a64a6a0cb0f1b92a222fb WatchSource:0}: Error finding container 36ea6fefe0d077b06ba9f96ea3e7a816fa9d4a8c8e2a64a6a0cb0f1b92a222fb: Status 404 returned error can't find the container with id 36ea6fefe0d077b06ba9f96ea3e7a816fa9d4a8c8e2a64a6a0cb0f1b92a222fb Jul 20 22:54:18.254189 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.254150 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" event={"ID":"9c326231-6e02-4973-aa69-116110666151","Type":"ContainerStarted","Data":"18db17b884da653daed96002342f81bff36e99d514c286a27e008b65d1b484fe"} Jul 20 22:54:18.255592 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.255574 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 22:54:18.255859 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.255838 2574 scope.go:117] "RemoveContainer" containerID="0f9b6446ae51fea4f99d956292dcf8255f3f3b44fef45b5ac0ea25802c0081e1" Jul 20 22:54:18.256020 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:18.255998 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-ncfsk_openshift-console-operator(41a5f424-012a-41c7-ad86-424296e7b06d)\"" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" podUID="41a5f424-012a-41c7-ad86-424296e7b06d" Jul 20 22:54:18.256779 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.256760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" event={"ID":"86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5","Type":"ContainerStarted","Data":"36ea6fefe0d077b06ba9f96ea3e7a816fa9d4a8c8e2a64a6a0cb0f1b92a222fb"} Jul 20 22:54:18.274862 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.274822 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bsqhr" podStartSLOduration=8.268582164 podStartE2EDuration="47.274808698s" totalImagesPullingTime="39.006226534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:18.273403098 +0000 UTC m=+47.769857522" watchObservedRunningTime="2026-07-20 22:54:18.274808698 +0000 UTC m=+47.771263149" Jul 20 22:54:18.897307 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.897271 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-ltbd4"] Jul 20 22:54:18.922711 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.922667 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-ltbd4"] Jul 20 22:54:18.922842 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.922797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:18.925024 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.924996 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 20 22:54:18.925187 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.925165 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vdw2j\"" Jul 20 22:54:18.925266 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.925240 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 20 22:54:18.925266 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.925250 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 20 22:54:18.925767 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:18.925750 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 20 22:54:19.044453 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.044422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-signing-key\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.044453 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.044454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwd2\" (UniqueName: \"kubernetes.io/projected/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-kube-api-access-qjwd2\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.044679 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.044485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-signing-cabundle\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.145566 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.145536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-signing-key\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.145698 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.145569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwd2\" (UniqueName: \"kubernetes.io/projected/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-kube-api-access-qjwd2\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.145698 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.145598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-signing-cabundle\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.146285 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.146266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-signing-cabundle\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.147825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.147779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-signing-key\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.153404 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.153375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwd2\" (UniqueName: \"kubernetes.io/projected/6dcc19b6-0f19-4810-8d92-4cb1e5100a28-kube-api-access-qjwd2\") pod \"service-ca-58598695f7-ltbd4\" (UID: \"6dcc19b6-0f19-4810-8d92-4cb1e5100a28\") " pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.231546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.231523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-ltbd4" Jul 20 22:54:19.263889 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.263854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" event={"ID":"86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5","Type":"ContainerStarted","Data":"fd3299e50db8807e6a5bdbe2a7fe16a9cd7fbab31cb53c335925af1ff1d12ed9"} Jul 20 22:54:19.373818 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:19.373787 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-ltbd4"] Jul 20 22:54:19.378425 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:19.378392 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcc19b6_0f19_4810_8d92_4cb1e5100a28.slice/crio-f2993e1d1292f904e4511ea0b549589fcc412081d120465af35ae08a165141de WatchSource:0}: Error finding container f2993e1d1292f904e4511ea0b549589fcc412081d120465af35ae08a165141de: Status 404 returned error can't find the container with id f2993e1d1292f904e4511ea0b549589fcc412081d120465af35ae08a165141de Jul 20 22:54:20.269852 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:20.269812 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" event={"ID":"86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5","Type":"ContainerStarted","Data":"3e6d34c62842f232d62e2b119f57b0fbbeebc29b1eac4a7921bcab58d84091b3"} Jul 20 22:54:20.271312 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:20.271281 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-ltbd4" event={"ID":"6dcc19b6-0f19-4810-8d92-4cb1e5100a28","Type":"ContainerStarted","Data":"e574ee874db3e7137547573454901f2e8a07ad9be9b85e8ebdabf8fba2e8757c"} Jul 20 22:54:20.271468 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:20.271328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-ltbd4" event={"ID":"6dcc19b6-0f19-4810-8d92-4cb1e5100a28","Type":"ContainerStarted","Data":"f2993e1d1292f904e4511ea0b549589fcc412081d120465af35ae08a165141de"} Jul 20 22:54:20.288921 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:20.288872 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pdxhm" podStartSLOduration=2.473896239 podStartE2EDuration="4.288855475s" totalImagesPullingTime="1.814959236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:20.28811931 +0000 UTC m=+49.784573746" watchObservedRunningTime="2026-07-20 22:54:20.288855475 +0000 UTC m=+49.785309899" Jul 20 22:54:20.308458 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:20.308412 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-ltbd4" podStartSLOduration=2.308395457 podStartE2EDuration="2.308395457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:20.307633991 +0000 UTC m=+49.804088415" watchObservedRunningTime="2026-07-20 22:54:20.308395457 +0000 UTC m=+49.804849880" Jul 20 22:54:21.767935 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:21.767896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:21.768409 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:21.768077 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 22:54:21.768409 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:21.768107 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b46cc44c9-brbhm: secret "image-registry-tls" not found Jul 20 22:54:21.768409 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:21.768197 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls podName:bb3a9d52-6223-4a61-94c0-066567aa187f nodeName:}" failed. No retries permitted until 2026-07-20 22:54:37.768176947 +0000 UTC m=+67.264631354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls") pod "image-registry-b46cc44c9-brbhm" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f") : secret "image-registry-tls" not found Jul 20 22:54:21.868650 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:21.868612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:21.868837 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:21.868663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:21.868837 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:21.868805 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 20 22:54:21.868943 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:21.868884 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert podName:2e4c6d2b-cce8-4b70-af8d-8791605ee28e nodeName:}" failed. No retries permitted until 2026-07-20 22:54:37.868864419 +0000 UTC m=+67.365318825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-tgjs4" (UID: "2e4c6d2b-cce8-4b70-af8d-8791605ee28e") : secret "networking-console-plugin-cert" not found Jul 20 22:54:21.869320 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:21.869296 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 22:54:21.869454 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:21.869357 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls podName:cba5b0a1-8585-46be-9dad-bc766849f281 nodeName:}" failed. No retries permitted until 2026-07-20 22:54:37.869343278 +0000 UTC m=+67.365797684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls") pod "dns-default-lzh55" (UID: "cba5b0a1-8585-46be-9dad-bc766849f281") : secret "dns-default-metrics-tls" not found Jul 20 22:54:26.256515 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:26.256459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:26.257074 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:26.256547 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:26.257074 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:26.256900 2574 scope.go:117] "RemoveContainer" containerID="0f9b6446ae51fea4f99d956292dcf8255f3f3b44fef45b5ac0ea25802c0081e1" Jul 20 22:54:26.257074 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:26.257044 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-ncfsk_openshift-console-operator(41a5f424-012a-41c7-ad86-424296e7b06d)\"" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" podUID="41a5f424-012a-41c7-ad86-424296e7b06d" Jul 20 22:54:33.174457 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:33.174428 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sg4m5" Jul 20 22:54:35.680146 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.680093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:35.680146 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.680154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:35.682829 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.682804 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 22:54:35.682901 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.682803 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 22:54:35.692952 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.692920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39-original-pull-secret\") pod \"global-pull-secret-syncer-xl88f\" (UID: \"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39\") " pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:35.692952 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.692937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d4e33a0-0442-47b0-9ecf-e833f909a185-metrics-certs\") pod \"network-metrics-daemon-gw7vz\" (UID: \"2d4e33a0-0442-47b0-9ecf-e833f909a185\") " pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:35.780741 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.780701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:35.783196 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.783167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv8w\" (UniqueName: \"kubernetes.io/projected/fcc384bb-5ccb-4609-948b-88c41d22bb2a-kube-api-access-wxv8w\") pod \"network-check-target-88x7j\" (UID: \"fcc384bb-5ccb-4609-948b-88c41d22bb2a\") " pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:35.822127 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.822103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xl88f" Jul 20 22:54:35.832833 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.832812 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xr56f\"" Jul 20 22:54:35.840712 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.840693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:35.841126 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.841111 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d688j\"" Jul 20 22:54:35.849252 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.849229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gw7vz" Jul 20 22:54:35.958959 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.958932 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xl88f"] Jul 20 22:54:35.962669 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:35.962639 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca14ac6b_9250_4bfc_84b6_ceb7b92f7b39.slice/crio-e4025b4bde03f4e017dc6f24fc0a204248a4706fac4442fe21ea0fd5191ff623 WatchSource:0}: Error finding container e4025b4bde03f4e017dc6f24fc0a204248a4706fac4442fe21ea0fd5191ff623: Status 404 returned error can't find the container with id e4025b4bde03f4e017dc6f24fc0a204248a4706fac4442fe21ea0fd5191ff623 Jul 20 22:54:35.982887 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.982712 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gw7vz"] Jul 20 22:54:35.985247 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:35.985220 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4e33a0_0442_47b0_9ecf_e833f909a185.slice/crio-cae3634791fa4bee7aec4e394910c249ffca4f46e3599699cc1d2f13695e2aee WatchSource:0}: Error finding container cae3634791fa4bee7aec4e394910c249ffca4f46e3599699cc1d2f13695e2aee: Status 404 returned error can't find the container with id cae3634791fa4bee7aec4e394910c249ffca4f46e3599699cc1d2f13695e2aee Jul 20 22:54:35.999437 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:35.999413 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-88x7j"] Jul 20 22:54:36.003782 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:36.003757 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc384bb_5ccb_4609_948b_88c41d22bb2a.slice/crio-4dc529bb70d9a4310e7fe1027a7ebff100203fa57223476cb4076541623b80eb WatchSource:0}: Error finding container 4dc529bb70d9a4310e7fe1027a7ebff100203fa57223476cb4076541623b80eb: Status 404 returned error can't find the container with id 4dc529bb70d9a4310e7fe1027a7ebff100203fa57223476cb4076541623b80eb Jul 20 22:54:36.325535 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:36.325425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xl88f" event={"ID":"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39","Type":"ContainerStarted","Data":"e4025b4bde03f4e017dc6f24fc0a204248a4706fac4442fe21ea0fd5191ff623"} Jul 20 22:54:36.326424 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:36.326397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gw7vz" event={"ID":"2d4e33a0-0442-47b0-9ecf-e833f909a185","Type":"ContainerStarted","Data":"cae3634791fa4bee7aec4e394910c249ffca4f46e3599699cc1d2f13695e2aee"} Jul 20 22:54:36.327810 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:36.327786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-88x7j" event={"ID":"fcc384bb-5ccb-4609-948b-88c41d22bb2a","Type":"ContainerStarted","Data":"9ad04434e71af9f187a8a9002a32fbdf70768c886739a3710122c9afebdf0fde"} Jul 20 22:54:36.327894 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:36.327825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-88x7j" event={"ID":"fcc384bb-5ccb-4609-948b-88c41d22bb2a","Type":"ContainerStarted","Data":"4dc529bb70d9a4310e7fe1027a7ebff100203fa57223476cb4076541623b80eb"} Jul 20 22:54:36.327939 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:36.327924 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:54:36.343762 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:36.343722 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-88x7j" podStartSLOduration=65.343709611 podStartE2EDuration="1m5.343709611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:36.342749014 +0000 UTC m=+65.839203439" watchObservedRunningTime="2026-07-20 22:54:36.343709611 +0000 UTC m=+65.840164036" Jul 20 22:54:37.803403 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:37.803368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:37.806005 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:37.805975 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"image-registry-b46cc44c9-brbhm\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:37.904204 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:37.904158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:37.904369 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:37.904209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:37.907220 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:37.907195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba5b0a1-8585-46be-9dad-bc766849f281-metrics-tls\") pod \"dns-default-lzh55\" (UID: \"cba5b0a1-8585-46be-9dad-bc766849f281\") " pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:37.907360 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:37.907332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2e4c6d2b-cce8-4b70-af8d-8791605ee28e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-tgjs4\" (UID: \"2e4c6d2b-cce8-4b70-af8d-8791605ee28e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:38.004013 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.003980 2574 scope.go:117] "RemoveContainer" containerID="0f9b6446ae51fea4f99d956292dcf8255f3f3b44fef45b5ac0ea25802c0081e1" Jul 20 22:54:38.036274 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.036246 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w7p55\"" Jul 20 22:54:38.044266 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.044246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:38.091299 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.091270 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fzdrm\"" Jul 20 22:54:38.099397 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.099370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" Jul 20 22:54:38.155216 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.155141 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtmm9\"" Jul 20 22:54:38.165783 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.164582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:38.210627 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.210550 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b46cc44c9-brbhm"] Jul 20 22:54:38.215277 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:38.214520 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3a9d52_6223_4a61_94c0_066567aa187f.slice/crio-07dd48f079aa10d5b55542b6c46e2557fca22f11f853e13d568947618f17b828 WatchSource:0}: Error finding container 07dd48f079aa10d5b55542b6c46e2557fca22f11f853e13d568947618f17b828: Status 404 returned error can't find the container with id 07dd48f079aa10d5b55542b6c46e2557fca22f11f853e13d568947618f17b828 Jul 20 22:54:38.267936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.267664 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4"] Jul 20 22:54:38.276556 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:38.276511 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4c6d2b_cce8_4b70_af8d_8791605ee28e.slice/crio-78f51c72363b3ef6245285646339fee134bfc4276d5ab6d23bbb8ce59a07da95 WatchSource:0}: Error finding container 78f51c72363b3ef6245285646339fee134bfc4276d5ab6d23bbb8ce59a07da95: Status 404 returned error can't find the container with id 78f51c72363b3ef6245285646339fee134bfc4276d5ab6d23bbb8ce59a07da95 Jul 20 22:54:38.331710 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.331634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lzh55"] Jul 20 22:54:38.336015 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:38.335982 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba5b0a1_8585_46be_9dad_bc766849f281.slice/crio-272dfbcc60bfda6c26d2b88713171039e832cecccd9dab54a1daba756596c350 WatchSource:0}: Error finding container 272dfbcc60bfda6c26d2b88713171039e832cecccd9dab54a1daba756596c350: Status 404 returned error can't find the container with id 272dfbcc60bfda6c26d2b88713171039e832cecccd9dab54a1daba756596c350 Jul 20 22:54:38.337881 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.337556 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gw7vz" event={"ID":"2d4e33a0-0442-47b0-9ecf-e833f909a185","Type":"ContainerStarted","Data":"53590f57b1dcf1864cb1ceaf86f3bc2f4d9d36f2eacae0bc946af616112fff88"} Jul 20 22:54:38.337881 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.337615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gw7vz" event={"ID":"2d4e33a0-0442-47b0-9ecf-e833f909a185","Type":"ContainerStarted","Data":"83076184bbf717af120f4cc4faf02582651fea113c4216fa65392f0ce8177c3d"} Jul 20 22:54:38.339140 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.339091 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" event={"ID":"2e4c6d2b-cce8-4b70-af8d-8791605ee28e","Type":"ContainerStarted","Data":"78f51c72363b3ef6245285646339fee134bfc4276d5ab6d23bbb8ce59a07da95"} Jul 20 22:54:38.340869 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.340845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" event={"ID":"bb3a9d52-6223-4a61-94c0-066567aa187f","Type":"ContainerStarted","Data":"4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917"} Jul 20 22:54:38.340957 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.340888 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" event={"ID":"bb3a9d52-6223-4a61-94c0-066567aa187f","Type":"ContainerStarted","Data":"07dd48f079aa10d5b55542b6c46e2557fca22f11f853e13d568947618f17b828"} Jul 20 22:54:38.341069 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.341048 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:38.343024 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.343002 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 22:54:38.343120 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.343056 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" event={"ID":"41a5f424-012a-41c7-ad86-424296e7b06d","Type":"ContainerStarted","Data":"6ce6739bd897c613db1391f1b522a2788f0c6ef726fe5d62cf93ebee3e62c718"} Jul 20 22:54:38.343322 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.343275 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:38.353849 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.353798 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gw7vz" podStartSLOduration=66.084039446 podStartE2EDuration="1m7.353779919s" totalImagesPullingTime="1.269740473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:38.352182321 +0000 UTC m=+67.848636745" watchObservedRunningTime="2026-07-20 22:54:38.353779919 +0000 UTC m=+67.850234344" Jul 20 22:54:38.369361 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.369313 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" podStartSLOduration=67.369301252 podStartE2EDuration="1m7.369301252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:38.368892957 +0000 UTC m=+67.865347382" watchObservedRunningTime="2026-07-20 22:54:38.369301252 +0000 UTC m=+67.865755676" Jul 20 22:54:38.385413 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.385358 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" podStartSLOduration=56.801206928 podStartE2EDuration="1m5.385338534s" totalImagesPullingTime="8.584131606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:38.384819332 +0000 UTC m=+67.881273758" watchObservedRunningTime="2026-07-20 22:54:38.385338534 +0000 UTC m=+67.881792959" Jul 20 22:54:38.571663 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:38.571619 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-ncfsk" Jul 20 22:54:39.349271 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.349233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lzh55" event={"ID":"cba5b0a1-8585-46be-9dad-bc766849f281","Type":"ContainerStarted","Data":"272dfbcc60bfda6c26d2b88713171039e832cecccd9dab54a1daba756596c350"} Jul 20 22:54:39.691819 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.691734 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf"] Jul 20 22:54:39.695139 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.695102 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:39.698301 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.698275 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 22:54:39.699547 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.699337 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.699547 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.699414 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.699547 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.699506 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rq2hq\"" Jul 20 22:54:39.700944 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.700920 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 20 22:54:39.712112 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.712087 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf"] Jul 20 22:54:39.720584 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.720560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2ae8192-2ca5-451b-91a6-0d4f033fd5a5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cddbb756-v5jlf\" (UID: \"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:39.720716 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.720597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdkn\" (UniqueName: \"kubernetes.io/projected/c2ae8192-2ca5-451b-91a6-0d4f033fd5a5-kube-api-access-sjdkn\") pod \"managed-serviceaccount-addon-agent-cddbb756-v5jlf\" (UID: \"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:39.753166 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.753131 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-d76hb"] Jul 20 22:54:39.756465 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.756439 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-flgnm"] Jul 20 22:54:39.756638 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.756615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-d76hb" Jul 20 22:54:39.759989 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.759966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.760608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.760463 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.760608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.760515 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.760608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.760566 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pzlsq\"" Jul 20 22:54:39.761657 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.761636 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx"] Jul 20 22:54:39.762241 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.762224 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 20 22:54:39.762336 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.762306 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m8m2n\"" Jul 20 22:54:39.763039 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.763019 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.763146 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.763067 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.763351 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.763335 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 20 22:54:39.765226 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.765208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:39.767861 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.767532 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 20 22:54:39.767861 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.767552 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 20 22:54:39.767861 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.767790 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:54:39.768695 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.768662 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8pnx6\"" Jul 20 22:54:39.770375 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.770348 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 20 22:54:39.772709 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.772452 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-d76hb"] Jul 20 22:54:39.774048 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.774026 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-flgnm"] Jul 20 22:54:39.781608 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.781583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx"] Jul 20 22:54:39.822736 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sdcwx\" (UID: \"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:39.822736 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.822999 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-tmp\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.822999 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-serving-cert\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.822999 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2ae8192-2ca5-451b-91a6-0d4f033fd5a5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cddbb756-v5jlf\" (UID: \"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:39.822999 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdkn\" (UniqueName: \"kubernetes.io/projected/c2ae8192-2ca5-451b-91a6-0d4f033fd5a5-kube-api-access-sjdkn\") pod \"managed-serviceaccount-addon-agent-cddbb756-v5jlf\" (UID: \"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:39.822999 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbxv\" (UniqueName: \"kubernetes.io/projected/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-kube-api-access-pbbxv\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.822999 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.822999 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgmd\" (UniqueName: \"kubernetes.io/projected/87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd-kube-api-access-gfgmd\") pod \"cluster-samples-operator-79b7c55f87-sdcwx\" (UID: \"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:39.823226 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.823026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-service-ca-bundle\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.823226 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.823055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qd8\" (UniqueName: \"kubernetes.io/projected/842f43cf-14e2-4ec6-b599-eedac3ed6b1a-kube-api-access-b8qd8\") pod \"downloads-5bc67dcd46-d76hb\" (UID: \"842f43cf-14e2-4ec6-b599-eedac3ed6b1a\") " pod="openshift-console/downloads-5bc67dcd46-d76hb" Jul 20 22:54:39.823226 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.823090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-snapshots\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.825928 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.825903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2ae8192-2ca5-451b-91a6-0d4f033fd5a5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cddbb756-v5jlf\" (UID: \"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:39.835826 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.835800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdkn\" (UniqueName: \"kubernetes.io/projected/c2ae8192-2ca5-451b-91a6-0d4f033fd5a5-kube-api-access-sjdkn\") pod \"managed-serviceaccount-addon-agent-cddbb756-v5jlf\" (UID: \"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:39.923462 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbxv\" (UniqueName: \"kubernetes.io/projected/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-kube-api-access-pbbxv\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.923633 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgmd\" (UniqueName: \"kubernetes.io/projected/87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd-kube-api-access-gfgmd\") pod \"cluster-samples-operator-79b7c55f87-sdcwx\" (UID: \"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:39.923633 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-service-ca-bundle\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.923633 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qd8\" (UniqueName: \"kubernetes.io/projected/842f43cf-14e2-4ec6-b599-eedac3ed6b1a-kube-api-access-b8qd8\") pod \"downloads-5bc67dcd46-d76hb\" (UID: \"842f43cf-14e2-4ec6-b599-eedac3ed6b1a\") " pod="openshift-console/downloads-5bc67dcd46-d76hb" Jul 20 22:54:39.923633 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-snapshots\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.923633 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sdcwx\" (UID: \"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:39.923916 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.923916 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-tmp\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.923916 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.923701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-serving-cert\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.924316 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.924269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-tmp\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.924316 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.924298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-service-ca-bundle\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.924485 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.924326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-snapshots\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.924576 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.924563 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.926481 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.926457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sdcwx\" (UID: \"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:39.927072 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.927034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-serving-cert\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.940556 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.940530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qd8\" (UniqueName: \"kubernetes.io/projected/842f43cf-14e2-4ec6-b599-eedac3ed6b1a-kube-api-access-b8qd8\") pod \"downloads-5bc67dcd46-d76hb\" (UID: \"842f43cf-14e2-4ec6-b599-eedac3ed6b1a\") " pod="openshift-console/downloads-5bc67dcd46-d76hb" Jul 20 22:54:39.947396 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.947351 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbxv\" (UniqueName: \"kubernetes.io/projected/b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88-kube-api-access-pbbxv\") pod \"insights-operator-555d9d7885-flgnm\" (UID: \"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88\") " pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:39.958095 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:39.958076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgmd\" (UniqueName: \"kubernetes.io/projected/87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd-kube-api-access-gfgmd\") pod \"cluster-samples-operator-79b7c55f87-sdcwx\" (UID: \"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:40.018643 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:40.018607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" Jul 20 22:54:40.070309 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:40.070272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-d76hb" Jul 20 22:54:40.081751 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:40.081727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-flgnm" Jul 20 22:54:40.088505 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:40.088475 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" Jul 20 22:54:41.437336 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:41.437167 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx"] Jul 20 22:54:41.449812 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:41.449762 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf"] Jul 20 22:54:41.456837 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:41.456808 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ae8192_2ca5_451b_91a6_0d4f033fd5a5.slice/crio-7eac77784b94451a4b341c8bff394cc700d69c372965b0f717c37215bbee44dc WatchSource:0}: Error finding container 7eac77784b94451a4b341c8bff394cc700d69c372965b0f717c37215bbee44dc: Status 404 returned error can't find the container with id 7eac77784b94451a4b341c8bff394cc700d69c372965b0f717c37215bbee44dc Jul 20 22:54:41.478053 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:41.477968 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-flgnm"] Jul 20 22:54:41.497116 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:41.497091 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-d76hb"] Jul 20 22:54:41.501747 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:41.501710 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842f43cf_14e2_4ec6_b599_eedac3ed6b1a.slice/crio-b3856fc53f5277ec45caae6d40e6848713847264c7747bccd02ae2414d8a517e WatchSource:0}: Error finding container b3856fc53f5277ec45caae6d40e6848713847264c7747bccd02ae2414d8a517e: Status 404 returned error can't find the container with id b3856fc53f5277ec45caae6d40e6848713847264c7747bccd02ae2414d8a517e Jul 20 22:54:42.383395 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.383321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xl88f" event={"ID":"ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39","Type":"ContainerStarted","Data":"4e5192710f5c386bb2b636e51be84f221c1e72aeefce36c3dd4c5f7829d007a4"} Jul 20 22:54:42.386413 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.385785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lzh55" event={"ID":"cba5b0a1-8585-46be-9dad-bc766849f281","Type":"ContainerStarted","Data":"5398c3a3abc62855a1b698b0e1c9702d5f5b606d0cc91ba3a817e633fafb2b10"} Jul 20 22:54:42.386413 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.385835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lzh55" event={"ID":"cba5b0a1-8585-46be-9dad-bc766849f281","Type":"ContainerStarted","Data":"73e4140de0809507b7ee129ab6ac98a226592983cb73ab17092697f1d497fccf"} Jul 20 22:54:42.386413 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.386346 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:42.389011 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.388953 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-d76hb" event={"ID":"842f43cf-14e2-4ec6-b599-eedac3ed6b1a","Type":"ContainerStarted","Data":"b3856fc53f5277ec45caae6d40e6848713847264c7747bccd02ae2414d8a517e"} Jul 20 22:54:42.392244 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.391838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" event={"ID":"2e4c6d2b-cce8-4b70-af8d-8791605ee28e","Type":"ContainerStarted","Data":"14217f76991f1eed2e937bc76bb901040e0a76b434d20a85f90b87e59df23c3b"} Jul 20 22:54:42.401154 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.400993 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xl88f" podStartSLOduration=66.121686424 podStartE2EDuration="1m11.400976944s" totalImagesPullingTime="5.27929052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:42.399169093 +0000 UTC m=+71.895623520" watchObservedRunningTime="2026-07-20 22:54:42.400976944 +0000 UTC m=+71.897431370" Jul 20 22:54:42.404897 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.404816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" event={"ID":"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd","Type":"ContainerStarted","Data":"a79173f287412f87b9c9e241c2c5046b64704c2d3c7aa1762c560ec03cba3cf5"} Jul 20 22:54:42.406425 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.406380 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" event={"ID":"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5","Type":"ContainerStarted","Data":"7eac77784b94451a4b341c8bff394cc700d69c372965b0f717c37215bbee44dc"} Jul 20 22:54:42.407874 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.407847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-flgnm" event={"ID":"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88","Type":"ContainerStarted","Data":"7dacd8872d4e56a7781ddffc0e45e793dc347958da4309e666ece0826e37c64c"} Jul 20 22:54:42.420343 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:42.419860 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lzh55" podStartSLOduration=33.515364145 podStartE2EDuration="36.419841382s" totalImagesPullingTime="2.904477237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:42.418579312 +0000 UTC m=+71.915033736" watchObservedRunningTime="2026-07-20 22:54:42.419841382 +0000 UTC m=+71.916295807" Jul 20 22:54:47.120861 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.120041 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-tgjs4" podStartSLOduration=39.156578318 podStartE2EDuration="42.120019146s" totalImagesPullingTime="2.963440828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:42.43731959 +0000 UTC m=+71.933774015" watchObservedRunningTime="2026-07-20 22:54:47.120019146 +0000 UTC m=+76.616473571" Jul 20 22:54:47.120861 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.120261 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x2dbp"] Jul 20 22:54:47.154446 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.154408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x2dbp"] Jul 20 22:54:47.154648 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.154592 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.156947 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.156756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 22:54:47.156947 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.156771 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 22:54:47.157156 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.157065 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lz6zc\"" Jul 20 22:54:47.182215 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.182191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.182215 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.182227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.182432 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.182255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e442cd5-5bdd-4c94-a841-738395c275cc-crio-socket\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.182432 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.182285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e442cd5-5bdd-4c94-a841-738395c275cc-data-volume\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.182432 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.182363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbg6\" (UniqueName: \"kubernetes.io/projected/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-api-access-nzbg6\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.283465 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.283425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.283654 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.283480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e442cd5-5bdd-4c94-a841-738395c275cc-crio-socket\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.283654 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.283577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e442cd5-5bdd-4c94-a841-738395c275cc-crio-socket\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.283654 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.283614 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:47.283816 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.283668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e442cd5-5bdd-4c94-a841-738395c275cc-data-volume\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.283816 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.283693 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:47.783669444 +0000 UTC m=+77.280123869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:47.283816 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.283747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbg6\" (UniqueName: \"kubernetes.io/projected/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-api-access-nzbg6\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.283816 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.283811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.284038 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.283896 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:47.284038 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.283948 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:47.783934545 +0000 UTC m=+77.280388967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : configmap "kube-rbac-proxy" not found Jul 20 22:54:47.284038 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.284006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e442cd5-5bdd-4c94-a841-738395c275cc-data-volume\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.295327 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.295294 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbg6\" (UniqueName: \"kubernetes.io/projected/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-api-access-nzbg6\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.433205 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.432914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" event={"ID":"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd","Type":"ContainerStarted","Data":"4e620b66475c930f0f611b9f05fc838ab0c37153a3852953174d9e789fe3989b"} Jul 20 22:54:47.433205 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.432976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" event={"ID":"87ca8360-1d2a-4ab3-8bc4-7e63dccf43bd","Type":"ContainerStarted","Data":"bc562502795f596fbe64687c5cdad7a7b3f8e6e7e9908bf37e96f61440490d93"} Jul 20 22:54:47.434561 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.434530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" event={"ID":"c2ae8192-2ca5-451b-91a6-0d4f033fd5a5","Type":"ContainerStarted","Data":"870e0fcb2db5d63ad87ed7c1abec3826e8def49e33af8c09909c07aaf89f7143"} Jul 20 22:54:47.436503 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.436468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-flgnm" event={"ID":"b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88","Type":"ContainerStarted","Data":"72d9a6f2c205e7ce9e805689b8714ce2ea9284baf771f53ee15464551b5fcc0d"} Jul 20 22:54:47.450101 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.450049 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sdcwx" podStartSLOduration=3.503917752 podStartE2EDuration="8.450029955s" totalImagesPullingTime="4.946112203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:47.448905943 +0000 UTC m=+76.945360369" watchObservedRunningTime="2026-07-20 22:54:47.450029955 +0000 UTC m=+76.946484380" Jul 20 22:54:47.464270 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.464209 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cddbb756-v5jlf" podStartSLOduration=3.48483467 podStartE2EDuration="8.464188183s" totalImagesPullingTime="4.979353513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:47.46297216 +0000 UTC m=+76.959426586" watchObservedRunningTime="2026-07-20 22:54:47.464188183 +0000 UTC m=+76.960642606" Jul 20 22:54:47.479977 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.479923 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-flgnm" podStartSLOduration=3.533237922 podStartE2EDuration="8.479904585s" totalImagesPullingTime="4.946666663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:47.478419583 +0000 UTC m=+76.974874008" watchObservedRunningTime="2026-07-20 22:54:47.479904585 +0000 UTC m=+76.976358986" Jul 20 22:54:47.788368 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.788261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.788575 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:47.788380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:47.788575 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.788397 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:47.788575 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.788463 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:47.788575 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.788475 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:48.788452161 +0000 UTC m=+78.284906564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:47.788575 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:47.788531 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:48.788514638 +0000 UTC m=+78.284969054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : configmap "kube-rbac-proxy" not found Jul 20 22:54:48.101146 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:48.101114 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lzh55_cba5b0a1-8585-46be-9dad-bc766849f281/dns/0.log" Jul 20 22:54:48.301684 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:48.301613 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lzh55_cba5b0a1-8585-46be-9dad-bc766849f281/kube-rbac-proxy/0.log" Jul 20 22:54:48.796259 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:48.796223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:48.796458 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:48.796408 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:48.796458 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:48.796422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:48.796625 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:48.796483 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:50.796461448 +0000 UTC m=+80.292915864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:48.796625 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:48.796535 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:48.796787 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:48.796705 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:50.79668737 +0000 UTC m=+80.293141776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : configmap "kube-rbac-proxy" not found Jul 20 22:54:49.301187 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:49.301156 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjsrr_feb100de-3890-47c2-868d-79a783d51b47/dns-node-resolver/0.log" Jul 20 22:54:49.701423 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:49.701383 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b46cc44c9-brbhm_bb3a9d52-6223-4a61-94c0-066567aa187f/registry/0.log" Jul 20 22:54:50.301067 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:50.301034 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vwqjp_6c0a5da1-61a9-4426-9646-800e40d3482c/node-ca/0.log" Jul 20 22:54:50.701742 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:50.701678 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jlh2g_efb0d356-51ab-4dd1-b549-084d6c826bd8/serve-healthcheck-canary/0.log" Jul 20 22:54:50.814082 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:50.814045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:50.814082 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:50.814092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:50.814315 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:50.814181 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 22:54:50.814315 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:50.814215 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 22:54:50.814315 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:50.814259 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:54.814237564 +0000 UTC m=+84.310691977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : configmap "kube-rbac-proxy" not found Jul 20 22:54:50.814315 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:50.814280 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls podName:5e442cd5-5bdd-4c94-a841-738395c275cc nodeName:}" failed. No retries permitted until 2026-07-20 22:54:54.81427113 +0000 UTC m=+84.310725541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x2dbp" (UID: "5e442cd5-5bdd-4c94-a841-738395c275cc") : secret "insights-runtime-extractor-tls" not found Jul 20 22:54:53.421558 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:53.421412 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lzh55" Jul 20 22:54:54.846872 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:54.846819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:54.847307 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:54.846891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:54.847432 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:54.847406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e442cd5-5bdd-4c94-a841-738395c275cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:54.849446 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:54.849419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e442cd5-5bdd-4c94-a841-738395c275cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x2dbp\" (UID: \"5e442cd5-5bdd-4c94-a841-738395c275cc\") " pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:54.965954 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:54.965922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x2dbp" Jul 20 22:54:58.049753 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.049690 2574 patch_prober.go:28] interesting pod/image-registry-b46cc44c9-brbhm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 20 22:54:58.050295 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.049793 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" podUID="bb3a9d52-6223-4a61-94c0-066567aa187f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 22:54:58.696922 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.696876 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rstkt"] Jul 20 22:54:58.705382 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.704191 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.709259 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.709214 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xfqr8\"" Jul 20 22:54:58.709840 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.709818 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 22:54:58.710032 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.710012 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 22:54:58.710531 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.710181 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 22:54:58.710531 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.710373 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 22:54:58.710531 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.710475 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 22:54:58.711001 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.710890 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.782794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fef0358d-b9a5-4a60-a522-01cf7860103a-metrics-client-ca\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.782861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.782895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpll\" (UniqueName: \"kubernetes.io/projected/fef0358d-b9a5-4a60-a522-01cf7860103a-kube-api-access-nqpll\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.782936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-sys\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.782965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-root\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.782986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-tls\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.783021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-textfile\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.783060 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-wtmp\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.783233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.783092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884118 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-sys\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884353 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-root\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884353 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-tls\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884353 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-textfile\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884353 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-wtmp\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884353 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884353 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fef0358d-b9a5-4a60-a522-01cf7860103a-metrics-client-ca\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884687 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-root\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884687 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884687 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpll\" (UniqueName: \"kubernetes.io/projected/fef0358d-b9a5-4a60-a522-01cf7860103a-kube-api-access-nqpll\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884687 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-sys\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884687 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.884560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-wtmp\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.884687 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:58.884639 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 20 22:54:58.884973 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:54:58.884698 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-tls podName:fef0358d-b9a5-4a60-a522-01cf7860103a nodeName:}" failed. No retries permitted until 2026-07-20 22:54:59.3846764 +0000 UTC m=+88.881130820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-tls") pod "node-exporter-rstkt" (UID: "fef0358d-b9a5-4a60-a522-01cf7860103a") : secret "node-exporter-tls" not found Jul 20 22:54:58.885586 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.885220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-textfile\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.885586 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.885479 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.885741 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.885702 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fef0358d-b9a5-4a60-a522-01cf7860103a-metrics-client-ca\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.888179 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.888131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:58.897188 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:58.897134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpll\" (UniqueName: \"kubernetes.io/projected/fef0358d-b9a5-4a60-a522-01cf7860103a-kube-api-access-nqpll\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:59.117262 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.117232 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x2dbp"] Jul 20 22:54:59.125853 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:59.125826 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e442cd5_5bdd_4c94_a841_738395c275cc.slice/crio-9af637020064393d4897a7361c02f2e9648f7a45913112acd0f605b905ca52a9 WatchSource:0}: Error finding container 9af637020064393d4897a7361c02f2e9648f7a45913112acd0f605b905ca52a9: Status 404 returned error can't find the container with id 9af637020064393d4897a7361c02f2e9648f7a45913112acd0f605b905ca52a9 Jul 20 22:54:59.354825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.354779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:54:59.389356 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.389320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-tls\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:59.392373 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.392343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fef0358d-b9a5-4a60-a522-01cf7860103a-node-exporter-tls\") pod \"node-exporter-rstkt\" (UID: \"fef0358d-b9a5-4a60-a522-01cf7860103a\") " pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:59.500927 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.500872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-d76hb" event={"ID":"842f43cf-14e2-4ec6-b599-eedac3ed6b1a","Type":"ContainerStarted","Data":"8d70d78721425ab07c0f8bc4b941d979008a8febbdad554b2c03b06b02db1e82"} Jul 20 22:54:59.501522 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.501240 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-d76hb" Jul 20 22:54:59.502512 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.502459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2dbp" event={"ID":"5e442cd5-5bdd-4c94-a841-738395c275cc","Type":"ContainerStarted","Data":"9af637020064393d4897a7361c02f2e9648f7a45913112acd0f605b905ca52a9"} Jul 20 22:54:59.514355 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.514325 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-d76hb" Jul 20 22:54:59.520367 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.520301 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-d76hb" podStartSLOduration=2.94235586 podStartE2EDuration="20.520278152s" totalImagesPullingTime="17.577922292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:54:59.518413492 +0000 UTC m=+89.014867910" watchObservedRunningTime="2026-07-20 22:54:59.520278152 +0000 UTC m=+89.016732578" Jul 20 22:54:59.619659 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:54:59.619563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rstkt" Jul 20 22:54:59.635777 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:54:59.635704 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef0358d_b9a5_4a60_a522_01cf7860103a.slice/crio-29f01b60269846ebda73be96c6fdf249dda130a9b6dec201b3442e68e08b64c2 WatchSource:0}: Error finding container 29f01b60269846ebda73be96c6fdf249dda130a9b6dec201b3442e68e08b64c2: Status 404 returned error can't find the container with id 29f01b60269846ebda73be96c6fdf249dda130a9b6dec201b3442e68e08b64c2 Jul 20 22:55:00.510648 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:00.510564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstkt" event={"ID":"fef0358d-b9a5-4a60-a522-01cf7860103a","Type":"ContainerStarted","Data":"29f01b60269846ebda73be96c6fdf249dda130a9b6dec201b3442e68e08b64c2"} Jul 20 22:55:01.728839 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:01.728802 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b46cc44c9-brbhm"] Jul 20 22:55:02.532275 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:02.532231 2574 generic.go:356] "Generic (PLEG): container finished" podID="fef0358d-b9a5-4a60-a522-01cf7860103a" containerID="c45982d18b33bd62459d615dc3554a1a91a5988728334e384dda809b77779f5a" exitCode=0 Jul 20 22:55:02.532462 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:02.532292 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstkt" event={"ID":"fef0358d-b9a5-4a60-a522-01cf7860103a","Type":"ContainerDied","Data":"c45982d18b33bd62459d615dc3554a1a91a5988728334e384dda809b77779f5a"} Jul 20 22:55:06.555220 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:06.555178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstkt" event={"ID":"fef0358d-b9a5-4a60-a522-01cf7860103a","Type":"ContainerStarted","Data":"6678f4f24a572f4c6c3d4d2383a0a694c84973eaba0072499bd5c6b471550bf4"} Jul 20 22:55:07.334848 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:07.334797 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-88x7j" Jul 20 22:55:07.563751 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:07.563657 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2dbp" event={"ID":"5e442cd5-5bdd-4c94-a841-738395c275cc","Type":"ContainerStarted","Data":"e656bfcc7f5df6190eeb5c2b3bbbab421c4c8fccf8ad5b43d003f458a5119bf9"} Jul 20 22:55:07.568874 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:07.567666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rstkt" event={"ID":"fef0358d-b9a5-4a60-a522-01cf7860103a","Type":"ContainerStarted","Data":"0ce45f31034ae511dfd10eacc31ff1f10d2f477fc715574648c351fa0a54622a"} Jul 20 22:55:07.588563 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:07.586602 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rstkt" podStartSLOduration=7.450002077 podStartE2EDuration="9.586580934s" totalImagesPullingTime="2.005300923s" totalInitContainerRuntime="131.277934ms" isStatefulPod=true podCreationTimestamp="2026-07-20 22:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:55:07.585985695 +0000 UTC m=+97.082440119" watchObservedRunningTime="2026-07-20 22:55:07.586580934 +0000 UTC m=+97.083035360" Jul 20 22:55:12.592120 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.592077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2dbp" event={"ID":"5e442cd5-5bdd-4c94-a841-738395c275cc","Type":"ContainerStarted","Data":"9e2ba7b63866a0ae4232c69af8d754a01a96efba33a7d5feb3e5d0217898d60c"} Jul 20 22:55:12.595041 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.595005 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57f5d6fc5c-drbzd"] Jul 20 22:55:12.614570 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.614510 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5d6fc5c-drbzd"] Jul 20 22:55:12.614748 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.614602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.617222 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.617196 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 22:55:12.617222 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.617207 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-x7rzq\"" Jul 20 22:55:12.617441 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.617207 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 22:55:12.617441 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.617249 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 22:55:12.618230 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.618077 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 22:55:12.618331 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.618267 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 22:55:12.623675 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.623474 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 20 22:55:12.717968 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.717932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdn7\" (UniqueName: \"kubernetes.io/projected/c1377fe6-c837-495d-b27a-eb5c58fe376f-kube-api-access-8zdn7\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.718177 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.717991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-service-ca\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.718177 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.718052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-trusted-ca-bundle\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.718177 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.718134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-serving-cert\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.718177 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.718170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-oauth-serving-cert\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.718364 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.718230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-config\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.718364 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.718275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-oauth-config\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.819575 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.819533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdn7\" (UniqueName: \"kubernetes.io/projected/c1377fe6-c837-495d-b27a-eb5c58fe376f-kube-api-access-8zdn7\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.819769 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.819603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-service-ca\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.819769 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.819629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-trusted-ca-bundle\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.819769 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.819658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-serving-cert\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.819769 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.819690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-oauth-serving-cert\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.819769 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.819733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-config\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.820025 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.819776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-oauth-config\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.820739 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.820710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-config\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.820868 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.820769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-trusted-ca-bundle\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.820984 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.820962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-service-ca\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.821265 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.821244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-oauth-serving-cert\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.822923 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.822904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-oauth-config\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.823090 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.823067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-serving-cert\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.829682 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.829660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdn7\" (UniqueName: \"kubernetes.io/projected/c1377fe6-c837-495d-b27a-eb5c58fe376f-kube-api-access-8zdn7\") pod \"console-57f5d6fc5c-drbzd\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:12.927251 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:12.927154 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:13.114969 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:13.114931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f5d6fc5c-drbzd"] Jul 20 22:55:13.119459 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:55:13.119430 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1377fe6_c837_495d_b27a_eb5c58fe376f.slice/crio-fdcd59955bc1f9de975aac07ec88616abfde7a4ed534ae591a2bdd786d6688a8 WatchSource:0}: Error finding container fdcd59955bc1f9de975aac07ec88616abfde7a4ed534ae591a2bdd786d6688a8: Status 404 returned error can't find the container with id fdcd59955bc1f9de975aac07ec88616abfde7a4ed534ae591a2bdd786d6688a8 Jul 20 22:55:13.598282 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:13.598226 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5d6fc5c-drbzd" event={"ID":"c1377fe6-c837-495d-b27a-eb5c58fe376f","Type":"ContainerStarted","Data":"fdcd59955bc1f9de975aac07ec88616abfde7a4ed534ae591a2bdd786d6688a8"} Jul 20 22:55:17.618454 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:17.618415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x2dbp" event={"ID":"5e442cd5-5bdd-4c94-a841-738395c275cc","Type":"ContainerStarted","Data":"114d3df45a14a340d91636e48dc18a6175214064fe60d7832f977bb6f3c31ef0"} Jul 20 22:55:17.620015 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:17.619991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5d6fc5c-drbzd" event={"ID":"c1377fe6-c837-495d-b27a-eb5c58fe376f","Type":"ContainerStarted","Data":"0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1"} Jul 20 22:55:17.636000 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:17.635951 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x2dbp" podStartSLOduration=12.652987326 podStartE2EDuration="30.635934416s" totalImagesPullingTime="17.98294709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 22:54:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:55:17.635240344 +0000 UTC m=+107.131694769" watchObservedRunningTime="2026-07-20 22:55:17.635934416 +0000 UTC m=+107.132388896" Jul 20 22:55:17.653512 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:17.653444 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57f5d6fc5c-drbzd" podStartSLOduration=1.418202008 podStartE2EDuration="5.653428159s" totalImagesPullingTime="4.235226151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:55:17.652359397 +0000 UTC m=+107.148813822" watchObservedRunningTime="2026-07-20 22:55:17.653428159 +0000 UTC m=+107.149882653" Jul 20 22:55:22.927597 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:22.927564 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:22.928063 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:22.927656 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:22.932858 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:22.932835 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:23.646765 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:23.646735 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:55:26.762423 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:26.762377 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" podUID="bb3a9d52-6223-4a61-94c0-066567aa187f" containerName="registry" containerID="cri-o://4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917" gracePeriod=30 Jul 20 22:55:27.039674 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.039601 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:55:27.145511 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145469 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvmp4\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-kube-api-access-zvmp4\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.145685 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145558 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-trusted-ca\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.145685 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145658 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-bound-sa-token\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.145758 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145702 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb3a9d52-6223-4a61-94c0-066567aa187f-ca-trust-extracted\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.145758 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145738 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-certificates\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.145829 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145768 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-installation-pull-secrets\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.145829 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145815 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.145923 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.145850 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-image-registry-private-configuration\") pod \"bb3a9d52-6223-4a61-94c0-066567aa187f\" (UID: \"bb3a9d52-6223-4a61-94c0-066567aa187f\") " Jul 20 22:55:27.146527 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.146056 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:55:27.146527 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.146159 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-trusted-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.146527 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.146206 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:55:27.148467 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.148436 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:55:27.148467 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.148452 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:55:27.148687 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.148592 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:55:27.148687 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.148651 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-kube-api-access-zvmp4" (OuterVolumeSpecName: "kube-api-access-zvmp4") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "kube-api-access-zvmp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:55:27.148756 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.148710 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:55:27.154898 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.154874 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb3a9d52-6223-4a61-94c0-066567aa187f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bb3a9d52-6223-4a61-94c0-066567aa187f" (UID: "bb3a9d52-6223-4a61-94c0-066567aa187f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:55:27.246805 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.246764 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-installation-pull-secrets\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.246805 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.246799 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-tls\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.246805 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.246810 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb3a9d52-6223-4a61-94c0-066567aa187f-image-registry-private-configuration\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.247031 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.246819 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmp4\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-kube-api-access-zvmp4\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.247031 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.246828 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb3a9d52-6223-4a61-94c0-066567aa187f-bound-sa-token\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.247031 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.246837 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb3a9d52-6223-4a61-94c0-066567aa187f-ca-trust-extracted\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.247031 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.246845 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb3a9d52-6223-4a61-94c0-066567aa187f-registry-certificates\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:55:27.658707 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.658668 2574 generic.go:356] "Generic (PLEG): container finished" podID="bb3a9d52-6223-4a61-94c0-066567aa187f" containerID="4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917" exitCode=0 Jul 20 22:55:27.658855 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.658716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" Jul 20 22:55:27.658855 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.658741 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" event={"ID":"bb3a9d52-6223-4a61-94c0-066567aa187f","Type":"ContainerDied","Data":"4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917"} Jul 20 22:55:27.658855 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.658799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b46cc44c9-brbhm" event={"ID":"bb3a9d52-6223-4a61-94c0-066567aa187f","Type":"ContainerDied","Data":"07dd48f079aa10d5b55542b6c46e2557fca22f11f853e13d568947618f17b828"} Jul 20 22:55:27.658855 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.658827 2574 scope.go:117] "RemoveContainer" containerID="4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917" Jul 20 22:55:27.670906 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.670885 2574 scope.go:117] "RemoveContainer" containerID="4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917" Jul 20 22:55:27.671198 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:55:27.671176 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917\": container with ID starting with 4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917 not found: ID does not exist" containerID="4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917" Jul 20 22:55:27.671238 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.671207 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917"} err="failed to get container status \"4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917\": rpc error: code = NotFound desc = could not find container \"4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917\": container with ID starting with 4f814dbeabc7cb3ab91f77ab73197bc284e976fb6733a9086f96e7a2900ca917 not found: ID does not exist" Jul 20 22:55:27.681407 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.681377 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b46cc44c9-brbhm"] Jul 20 22:55:27.685329 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:27.685306 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b46cc44c9-brbhm"] Jul 20 22:55:29.008363 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:29.008326 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3a9d52-6223-4a61-94c0-066567aa187f" path="/var/lib/kubelet/pods/bb3a9d52-6223-4a61-94c0-066567aa187f/volumes" Jul 20 22:55:31.676095 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:31.676057 2574 generic.go:356] "Generic (PLEG): container finished" podID="077d3a13-d78c-4672-84c2-7dd2a551f322" containerID="070ffbf3b83405fc0791169bc2a45d3786e9ba1cf2db58eac36b617379f4415e" exitCode=0 Jul 20 22:55:31.676528 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:31.676131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" event={"ID":"077d3a13-d78c-4672-84c2-7dd2a551f322","Type":"ContainerDied","Data":"070ffbf3b83405fc0791169bc2a45d3786e9ba1cf2db58eac36b617379f4415e"} Jul 20 22:55:31.676579 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:31.676562 2574 scope.go:117] "RemoveContainer" containerID="070ffbf3b83405fc0791169bc2a45d3786e9ba1cf2db58eac36b617379f4415e" Jul 20 22:55:32.682158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:32.682120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-84s25" event={"ID":"077d3a13-d78c-4672-84c2-7dd2a551f322","Type":"ContainerStarted","Data":"a3892560d67769193e3961d8affabee242f81fa1dfee95ca9bd702892ea3c4ca"} Jul 20 22:55:46.735897 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:46.735857 2574 generic.go:356] "Generic (PLEG): container finished" podID="c79080fd-58d2-4bec-a34c-476ef421391d" containerID="a091f097521675c1a379dfeb86b3d54f2975872d252370fff57a82c7b3c85557" exitCode=0 Jul 20 22:55:46.736335 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:46.735933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" event={"ID":"c79080fd-58d2-4bec-a34c-476ef421391d","Type":"ContainerDied","Data":"a091f097521675c1a379dfeb86b3d54f2975872d252370fff57a82c7b3c85557"} Jul 20 22:55:46.736335 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:46.736258 2574 scope.go:117] "RemoveContainer" containerID="a091f097521675c1a379dfeb86b3d54f2975872d252370fff57a82c7b3c85557" Jul 20 22:55:47.741744 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:55:47.741708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-6rh4m" event={"ID":"c79080fd-58d2-4bec-a34c-476ef421391d","Type":"ContainerStarted","Data":"bcc142d092f8df1e91e5fb2ab2c3ab9aaedd96b5fbb15accdc665ce934d5f340"} Jul 20 22:56:18.878310 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:18.878273 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68c95987-hk2tx"] Jul 20 22:56:18.878908 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:18.878765 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3a9d52-6223-4a61-94c0-066567aa187f" containerName="registry" Jul 20 22:56:18.878908 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:18.878788 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3a9d52-6223-4a61-94c0-066567aa187f" containerName="registry" Jul 20 22:56:18.878908 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:18.878875 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3a9d52-6223-4a61-94c0-066567aa187f" containerName="registry" Jul 20 22:56:18.899297 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:18.899270 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c95987-hk2tx"] Jul 20 22:56:18.899459 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:18.899369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.041131 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.041096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-serving-cert\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.041335 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.041156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-trusted-ca-bundle\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.041335 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.041175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzdd\" (UniqueName: \"kubernetes.io/projected/6d8e189e-e3a4-4a6a-afed-b4733564df78-kube-api-access-fmzdd\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.041335 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.041245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-service-ca\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.041335 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.041302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-config\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.041335 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.041325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-oauth-config\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.041587 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.041347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-oauth-serving-cert\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.142788 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.142695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-config\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.142788 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.142731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-oauth-config\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.142788 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.142753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-oauth-serving-cert\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.142788 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.142781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-serving-cert\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.143107 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.142846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-trusted-ca-bundle\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.143107 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.142873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzdd\" (UniqueName: \"kubernetes.io/projected/6d8e189e-e3a4-4a6a-afed-b4733564df78-kube-api-access-fmzdd\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.143107 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.142898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-service-ca\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.143617 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.143585 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-config\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.143728 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.143589 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-oauth-serving-cert\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.143728 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.143633 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-service-ca\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.143952 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.143933 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-trusted-ca-bundle\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.145954 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.145923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-oauth-config\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.146043 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.146001 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-serving-cert\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.151363 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.151339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzdd\" (UniqueName: \"kubernetes.io/projected/6d8e189e-e3a4-4a6a-afed-b4733564df78-kube-api-access-fmzdd\") pod \"console-68c95987-hk2tx\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.208427 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.208388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:19.350019 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.349994 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c95987-hk2tx"] Jul 20 22:56:19.352829 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:56:19.352797 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8e189e_e3a4_4a6a_afed_b4733564df78.slice/crio-4366b6d4ea6a5554ce6728272bbdecde10333b4d382d4fa04a75233f88e1e1a2 WatchSource:0}: Error finding container 4366b6d4ea6a5554ce6728272bbdecde10333b4d382d4fa04a75233f88e1e1a2: Status 404 returned error can't find the container with id 4366b6d4ea6a5554ce6728272bbdecde10333b4d382d4fa04a75233f88e1e1a2 Jul 20 22:56:19.862062 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.862019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c95987-hk2tx" event={"ID":"6d8e189e-e3a4-4a6a-afed-b4733564df78","Type":"ContainerStarted","Data":"2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725"} Jul 20 22:56:19.862244 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.862076 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c95987-hk2tx" event={"ID":"6d8e189e-e3a4-4a6a-afed-b4733564df78","Type":"ContainerStarted","Data":"4366b6d4ea6a5554ce6728272bbdecde10333b4d382d4fa04a75233f88e1e1a2"} Jul 20 22:56:19.880890 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:19.880829 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68c95987-hk2tx" podStartSLOduration=1.880814552 podStartE2EDuration="1.880814552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:56:19.880050555 +0000 UTC m=+169.376504980" watchObservedRunningTime="2026-07-20 22:56:19.880814552 +0000 UTC m=+169.377268976" Jul 20 22:56:29.209207 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:29.209103 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:29.209207 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:29.209185 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:29.214175 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:29.214146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:29.907784 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:29.907751 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68c95987-hk2tx" Jul 20 22:56:29.951866 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:29.951824 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f5d6fc5c-drbzd"] Jul 20 22:56:54.974515 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:54.974452 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57f5d6fc5c-drbzd" podUID="c1377fe6-c837-495d-b27a-eb5c58fe376f" containerName="console" containerID="cri-o://0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1" gracePeriod=15 Jul 20 22:56:55.223699 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.223671 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:56:55.330718 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.330622 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-trusted-ca-bundle\") pod \"c1377fe6-c837-495d-b27a-eb5c58fe376f\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " Jul 20 22:56:55.330718 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.330660 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-oauth-serving-cert\") pod \"c1377fe6-c837-495d-b27a-eb5c58fe376f\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " Jul 20 22:56:55.330718 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.330687 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zdn7\" (UniqueName: \"kubernetes.io/projected/c1377fe6-c837-495d-b27a-eb5c58fe376f-kube-api-access-8zdn7\") pod \"c1377fe6-c837-495d-b27a-eb5c58fe376f\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " Jul 20 22:56:55.330718 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.330713 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-config\") pod \"c1377fe6-c837-495d-b27a-eb5c58fe376f\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " Jul 20 22:56:55.331048 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.330750 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-serving-cert\") pod \"c1377fe6-c837-495d-b27a-eb5c58fe376f\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " Jul 20 22:56:55.331048 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.330770 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-oauth-config\") pod \"c1377fe6-c837-495d-b27a-eb5c58fe376f\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " Jul 20 22:56:55.331048 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.330798 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-service-ca\") pod \"c1377fe6-c837-495d-b27a-eb5c58fe376f\" (UID: \"c1377fe6-c837-495d-b27a-eb5c58fe376f\") " Jul 20 22:56:55.331236 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.331197 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c1377fe6-c837-495d-b27a-eb5c58fe376f" (UID: "c1377fe6-c837-495d-b27a-eb5c58fe376f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:56:55.331236 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.331195 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-config" (OuterVolumeSpecName: "console-config") pod "c1377fe6-c837-495d-b27a-eb5c58fe376f" (UID: "c1377fe6-c837-495d-b27a-eb5c58fe376f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:56:55.331334 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.331281 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-service-ca" (OuterVolumeSpecName: "service-ca") pod "c1377fe6-c837-495d-b27a-eb5c58fe376f" (UID: "c1377fe6-c837-495d-b27a-eb5c58fe376f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:56:55.331334 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.331319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c1377fe6-c837-495d-b27a-eb5c58fe376f" (UID: "c1377fe6-c837-495d-b27a-eb5c58fe376f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 22:56:55.333084 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.333060 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c1377fe6-c837-495d-b27a-eb5c58fe376f" (UID: "c1377fe6-c837-495d-b27a-eb5c58fe376f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:56:55.333173 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.333100 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1377fe6-c837-495d-b27a-eb5c58fe376f-kube-api-access-8zdn7" (OuterVolumeSpecName: "kube-api-access-8zdn7") pod "c1377fe6-c837-495d-b27a-eb5c58fe376f" (UID: "c1377fe6-c837-495d-b27a-eb5c58fe376f"). InnerVolumeSpecName "kube-api-access-8zdn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:56:55.333173 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.333100 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c1377fe6-c837-495d-b27a-eb5c58fe376f" (UID: "c1377fe6-c837-495d-b27a-eb5c58fe376f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 22:56:55.431476 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.431420 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-trusted-ca-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:56:55.431476 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.431468 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-oauth-serving-cert\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:56:55.431476 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.431478 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdn7\" (UniqueName: \"kubernetes.io/projected/c1377fe6-c837-495d-b27a-eb5c58fe376f-kube-api-access-8zdn7\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:56:55.431476 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.431511 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-config\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:56:55.431476 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.431521 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-serving-cert\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:56:55.431810 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.431530 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1377fe6-c837-495d-b27a-eb5c58fe376f-console-oauth-config\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:56:55.431810 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.431538 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1377fe6-c837-495d-b27a-eb5c58fe376f-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:56:55.997124 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.997088 2574 generic.go:356] "Generic (PLEG): container finished" podID="c1377fe6-c837-495d-b27a-eb5c58fe376f" containerID="0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1" exitCode=0 Jul 20 22:56:55.997609 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.997133 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f5d6fc5c-drbzd" Jul 20 22:56:55.997609 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.997149 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5d6fc5c-drbzd" event={"ID":"c1377fe6-c837-495d-b27a-eb5c58fe376f","Type":"ContainerDied","Data":"0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1"} Jul 20 22:56:55.997609 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.997194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f5d6fc5c-drbzd" event={"ID":"c1377fe6-c837-495d-b27a-eb5c58fe376f","Type":"ContainerDied","Data":"fdcd59955bc1f9de975aac07ec88616abfde7a4ed534ae591a2bdd786d6688a8"} Jul 20 22:56:55.997609 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:55.997215 2574 scope.go:117] "RemoveContainer" containerID="0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1" Jul 20 22:56:56.005805 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:56.005788 2574 scope.go:117] "RemoveContainer" containerID="0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1" Jul 20 22:56:56.006064 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:56:56.006042 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1\": container with ID starting with 0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1 not found: ID does not exist" containerID="0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1" Jul 20 22:56:56.006129 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:56.006078 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1"} err="failed to get container status \"0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1\": rpc error: code = NotFound desc = could not find container \"0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1\": container with ID starting with 0bb2b2941184de9da9169044ea66185019b204fcdc4bcd60d6434387925621c1 not found: ID does not exist" Jul 20 22:56:56.017445 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:56.017417 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f5d6fc5c-drbzd"] Jul 20 22:56:56.020975 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:56.020950 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57f5d6fc5c-drbzd"] Jul 20 22:56:57.010762 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:56:57.010719 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1377fe6-c837-495d-b27a-eb5c58fe376f" path="/var/lib/kubelet/pods/c1377fe6-c837-495d-b27a-eb5c58fe376f/volumes" Jul 20 22:57:42.081451 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.081414 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd"] Jul 20 22:57:42.081993 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.081927 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1377fe6-c837-495d-b27a-eb5c58fe376f" containerName="console" Jul 20 22:57:42.081993 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.081949 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1377fe6-c837-495d-b27a-eb5c58fe376f" containerName="console" Jul 20 22:57:42.082120 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.082029 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1377fe6-c837-495d-b27a-eb5c58fe376f" containerName="console" Jul 20 22:57:42.086239 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.086213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.089272 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.089241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s7wfd\"" Jul 20 22:57:42.089418 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.089382 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 22:57:42.090048 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.090031 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 22:57:42.097977 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.097931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd"] Jul 20 22:57:42.181403 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.181365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.181403 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.181406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swz8h\" (UniqueName: \"kubernetes.io/projected/ddb17e17-1519-47a2-aa41-716e36ec8733-kube-api-access-swz8h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.181638 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.181540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.282757 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.282714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swz8h\" (UniqueName: \"kubernetes.io/projected/ddb17e17-1519-47a2-aa41-716e36ec8733-kube-api-access-swz8h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.282941 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.282807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.282941 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.282859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.283196 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.283172 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.283265 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.283213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.293218 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.293184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swz8h\" (UniqueName: \"kubernetes.io/projected/ddb17e17-1519-47a2-aa41-716e36ec8733-kube-api-access-swz8h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.395634 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.395597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:57:42.531435 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:42.531254 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd"] Jul 20 22:57:42.534029 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:57:42.534000 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb17e17_1519_47a2_aa41_716e36ec8733.slice/crio-9d096a2f520e993c235e2737cdaa937019b0c60d00fdf3acb7ef6f7c660e4e35 WatchSource:0}: Error finding container 9d096a2f520e993c235e2737cdaa937019b0c60d00fdf3acb7ef6f7c660e4e35: Status 404 returned error can't find the container with id 9d096a2f520e993c235e2737cdaa937019b0c60d00fdf3acb7ef6f7c660e4e35 Jul 20 22:57:43.172636 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:43.172602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" event={"ID":"ddb17e17-1519-47a2-aa41-716e36ec8733","Type":"ContainerStarted","Data":"9d096a2f520e993c235e2737cdaa937019b0c60d00fdf3acb7ef6f7c660e4e35"} Jul 20 22:57:49.197212 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:49.197175 2574 generic.go:356] "Generic (PLEG): container finished" podID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerID="49716e1c4f4ade9ac74b2449998adaa16306f944d9fe77422452465d536bbb32" exitCode=0 Jul 20 22:57:49.197598 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:49.197233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" event={"ID":"ddb17e17-1519-47a2-aa41-716e36ec8733","Type":"ContainerDied","Data":"49716e1c4f4ade9ac74b2449998adaa16306f944d9fe77422452465d536bbb32"} Jul 20 22:57:52.212423 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:52.212389 2574 generic.go:356] "Generic (PLEG): container finished" podID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerID="d89c87b8177fcffd7d83aea862617d9a1edcea67124a9698e0f97e79d32287d6" exitCode=0 Jul 20 22:57:52.212851 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:52.212479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" event={"ID":"ddb17e17-1519-47a2-aa41-716e36ec8733","Type":"ContainerDied","Data":"d89c87b8177fcffd7d83aea862617d9a1edcea67124a9698e0f97e79d32287d6"} Jul 20 22:57:59.243591 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:59.243558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" event={"ID":"ddb17e17-1519-47a2-aa41-716e36ec8733","Type":"ContainerStarted","Data":"67a8e9a5ff5e29f5b99299baf43bc1e5f7d803ff277da92514487145f546899a"} Jul 20 22:57:59.261564 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:57:59.261515 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" podStartSLOduration=1.771664846 podStartE2EDuration="17.261476995s" totalImagesPullingTime="15.489812149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:57:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:57:59.259939638 +0000 UTC m=+268.756394065" watchObservedRunningTime="2026-07-20 22:57:59.261476995 +0000 UTC m=+268.757931418" Jul 20 22:58:00.249362 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:00.249328 2574 generic.go:356] "Generic (PLEG): container finished" podID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerID="67a8e9a5ff5e29f5b99299baf43bc1e5f7d803ff277da92514487145f546899a" exitCode=0 Jul 20 22:58:00.249362 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:00.249368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" event={"ID":"ddb17e17-1519-47a2-aa41-716e36ec8733","Type":"ContainerDied","Data":"67a8e9a5ff5e29f5b99299baf43bc1e5f7d803ff277da92514487145f546899a"} Jul 20 22:58:01.377927 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.377902 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:58:01.436310 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.436283 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-util\") pod \"ddb17e17-1519-47a2-aa41-716e36ec8733\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " Jul 20 22:58:01.436432 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.436329 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-bundle\") pod \"ddb17e17-1519-47a2-aa41-716e36ec8733\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " Jul 20 22:58:01.436432 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.436366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swz8h\" (UniqueName: \"kubernetes.io/projected/ddb17e17-1519-47a2-aa41-716e36ec8733-kube-api-access-swz8h\") pod \"ddb17e17-1519-47a2-aa41-716e36ec8733\" (UID: \"ddb17e17-1519-47a2-aa41-716e36ec8733\") " Jul 20 22:58:01.437133 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.437102 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-bundle" (OuterVolumeSpecName: "bundle") pod "ddb17e17-1519-47a2-aa41-716e36ec8733" (UID: "ddb17e17-1519-47a2-aa41-716e36ec8733"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:58:01.438472 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.438443 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb17e17-1519-47a2-aa41-716e36ec8733-kube-api-access-swz8h" (OuterVolumeSpecName: "kube-api-access-swz8h") pod "ddb17e17-1519-47a2-aa41-716e36ec8733" (UID: "ddb17e17-1519-47a2-aa41-716e36ec8733"). InnerVolumeSpecName "kube-api-access-swz8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:58:01.441207 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.441186 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-util" (OuterVolumeSpecName: "util") pod "ddb17e17-1519-47a2-aa41-716e36ec8733" (UID: "ddb17e17-1519-47a2-aa41-716e36ec8733"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:58:01.537825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.537749 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swz8h\" (UniqueName: \"kubernetes.io/projected/ddb17e17-1519-47a2-aa41-716e36ec8733-kube-api-access-swz8h\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:01.537825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.537775 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-util\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:01.537825 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:01.537785 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddb17e17-1519-47a2-aa41-716e36ec8733-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:02.258441 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:02.258414 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" Jul 20 22:58:02.258633 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:02.258412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754075vxd" event={"ID":"ddb17e17-1519-47a2-aa41-716e36ec8733","Type":"ContainerDied","Data":"9d096a2f520e993c235e2737cdaa937019b0c60d00fdf3acb7ef6f7c660e4e35"} Jul 20 22:58:02.258633 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:02.258551 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d096a2f520e993c235e2737cdaa937019b0c60d00fdf3acb7ef6f7c660e4e35" Jul 20 22:58:09.921295 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921261 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w"] Jul 20 22:58:09.921814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921565 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerName="util" Jul 20 22:58:09.921814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921576 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerName="util" Jul 20 22:58:09.921814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921583 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerName="pull" Jul 20 22:58:09.921814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921589 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerName="pull" Jul 20 22:58:09.921814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921601 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerName="extract" Jul 20 22:58:09.921814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921606 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerName="extract" Jul 20 22:58:09.921814 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.921652 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb17e17-1519-47a2-aa41-716e36ec8733" containerName="extract" Jul 20 22:58:09.925955 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.925939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:09.929249 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.929226 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:58:09.929785 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.929762 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lrb8z\"" Jul 20 22:58:09.929882 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.929782 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 20 22:58:09.943313 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.943288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w"] Jul 20 22:58:09.997937 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.997903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njk6t\" (UniqueName: \"kubernetes.io/projected/995b0346-d016-4e9a-84b3-b1668cae6e6f-kube-api-access-njk6t\") pod \"cert-manager-operator-controller-manager-55c444bf86-jc66w\" (UID: \"995b0346-d016-4e9a-84b3-b1668cae6e6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:09.998085 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:09.997962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/995b0346-d016-4e9a-84b3-b1668cae6e6f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jc66w\" (UID: \"995b0346-d016-4e9a-84b3-b1668cae6e6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:10.098833 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:10.098804 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njk6t\" (UniqueName: \"kubernetes.io/projected/995b0346-d016-4e9a-84b3-b1668cae6e6f-kube-api-access-njk6t\") pod \"cert-manager-operator-controller-manager-55c444bf86-jc66w\" (UID: \"995b0346-d016-4e9a-84b3-b1668cae6e6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:10.098986 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:10.098863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/995b0346-d016-4e9a-84b3-b1668cae6e6f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jc66w\" (UID: \"995b0346-d016-4e9a-84b3-b1668cae6e6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:10.099190 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:10.099175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/995b0346-d016-4e9a-84b3-b1668cae6e6f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-jc66w\" (UID: \"995b0346-d016-4e9a-84b3-b1668cae6e6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:10.107727 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:10.107700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njk6t\" (UniqueName: \"kubernetes.io/projected/995b0346-d016-4e9a-84b3-b1668cae6e6f-kube-api-access-njk6t\") pod \"cert-manager-operator-controller-manager-55c444bf86-jc66w\" (UID: \"995b0346-d016-4e9a-84b3-b1668cae6e6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:10.235206 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:10.235103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" Jul 20 22:58:10.375936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:10.375901 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w"] Jul 20 22:58:10.379901 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:58:10.379874 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995b0346_d016_4e9a_84b3_b1668cae6e6f.slice/crio-ea769ee2038292f3af72bd1eefeac73e6a25bccb69ac5577e1aee4b6381b85f8 WatchSource:0}: Error finding container ea769ee2038292f3af72bd1eefeac73e6a25bccb69ac5577e1aee4b6381b85f8: Status 404 returned error can't find the container with id ea769ee2038292f3af72bd1eefeac73e6a25bccb69ac5577e1aee4b6381b85f8 Jul 20 22:58:11.295882 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:11.295825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" event={"ID":"995b0346-d016-4e9a-84b3-b1668cae6e6f","Type":"ContainerStarted","Data":"ea769ee2038292f3af72bd1eefeac73e6a25bccb69ac5577e1aee4b6381b85f8"} Jul 20 22:58:12.302620 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:12.302567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" event={"ID":"995b0346-d016-4e9a-84b3-b1668cae6e6f","Type":"ContainerStarted","Data":"8e5ec9fb54f28d6fbba0e67ab84a81700897c171f81572f1e2b6602b526380eb"} Jul 20 22:58:14.348413 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.348355 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-jc66w" podStartSLOduration=3.9010990960000003 podStartE2EDuration="5.348339202s" totalImagesPullingTime="1.447240106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:58:12.330122779 +0000 UTC m=+281.826577204" watchObservedRunningTime="2026-07-20 22:58:14.348339202 +0000 UTC m=+283.844793626" Jul 20 22:58:14.349853 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.349830 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf"] Jul 20 22:58:14.353354 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.353339 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.355601 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.355584 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s7wfd\"" Jul 20 22:58:14.355675 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.355606 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 22:58:14.356400 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.356386 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 22:58:14.361358 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.361338 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf"] Jul 20 22:58:14.432331 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.432299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.432519 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.432339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnsd\" (UniqueName: \"kubernetes.io/projected/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-kube-api-access-ptnsd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.432519 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.432368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.532740 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.532707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.532740 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.532754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnsd\" (UniqueName: \"kubernetes.io/projected/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-kube-api-access-ptnsd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.532981 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.532776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.533168 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.533150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.533226 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.533176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.541896 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.541876 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnsd\" (UniqueName: \"kubernetes.io/projected/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-kube-api-access-ptnsd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.662996 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.662909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:14.791129 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:14.791091 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf"] Jul 20 22:58:14.793134 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:58:14.793108 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061e9348_13bd_4876_a3ec_0d5ebbe3dedc.slice/crio-e0afbaf82618dc8e1812bfecb1f02001f12c1716093e1957e0c17785d25edf76 WatchSource:0}: Error finding container e0afbaf82618dc8e1812bfecb1f02001f12c1716093e1957e0c17785d25edf76: Status 404 returned error can't find the container with id e0afbaf82618dc8e1812bfecb1f02001f12c1716093e1957e0c17785d25edf76 Jul 20 22:58:15.315593 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:15.315555 2574 generic.go:356] "Generic (PLEG): container finished" podID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerID="d8bd7aa16a62b3e39d4e4c084c280ed2c3ceb2420157447d1b894ba98ba3c2af" exitCode=0 Jul 20 22:58:15.315845 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:15.315637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" event={"ID":"061e9348-13bd-4876-a3ec-0d5ebbe3dedc","Type":"ContainerDied","Data":"d8bd7aa16a62b3e39d4e4c084c280ed2c3ceb2420157447d1b894ba98ba3c2af"} Jul 20 22:58:15.315845 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:15.315689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" event={"ID":"061e9348-13bd-4876-a3ec-0d5ebbe3dedc","Type":"ContainerStarted","Data":"e0afbaf82618dc8e1812bfecb1f02001f12c1716093e1957e0c17785d25edf76"} Jul 20 22:58:17.327477 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:17.327441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" event={"ID":"061e9348-13bd-4876-a3ec-0d5ebbe3dedc","Type":"ContainerStarted","Data":"31c982a74aebade92bda414980062b8e02f2631e8910bcaba4f654ab426e5ad6"} Jul 20 22:58:18.333853 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:18.333816 2574 generic.go:356] "Generic (PLEG): container finished" podID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerID="31c982a74aebade92bda414980062b8e02f2631e8910bcaba4f654ab426e5ad6" exitCode=0 Jul 20 22:58:18.334232 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:18.333905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" event={"ID":"061e9348-13bd-4876-a3ec-0d5ebbe3dedc","Type":"ContainerDied","Data":"31c982a74aebade92bda414980062b8e02f2631e8910bcaba4f654ab426e5ad6"} Jul 20 22:58:19.244777 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.244746 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mz47t"] Jul 20 22:58:19.247912 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.247891 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.250284 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.250260 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-s2wpd\"" Jul 20 22:58:19.250394 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.250266 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 20 22:58:19.251059 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.251043 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 20 22:58:19.255468 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.255445 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mz47t"] Jul 20 22:58:19.340354 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.340323 2574 generic.go:356] "Generic (PLEG): container finished" podID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerID="b5a3aaa227d0401f361ebe1a190ab98281a5900a576faf5126f46b49580ad476" exitCode=0 Jul 20 22:58:19.340755 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.340412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" event={"ID":"061e9348-13bd-4876-a3ec-0d5ebbe3dedc","Type":"ContainerDied","Data":"b5a3aaa227d0401f361ebe1a190ab98281a5900a576faf5126f46b49580ad476"} Jul 20 22:58:19.377418 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.377385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19640324-5142-4329-b8a4-63db6286cc9f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mz47t\" (UID: \"19640324-5142-4329-b8a4-63db6286cc9f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.377596 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.377425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-499wj\" (UniqueName: \"kubernetes.io/projected/19640324-5142-4329-b8a4-63db6286cc9f-kube-api-access-499wj\") pod \"cert-manager-cainjector-559cdd8c87-mz47t\" (UID: \"19640324-5142-4329-b8a4-63db6286cc9f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.478862 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.478828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19640324-5142-4329-b8a4-63db6286cc9f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mz47t\" (UID: \"19640324-5142-4329-b8a4-63db6286cc9f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.478862 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.478868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-499wj\" (UniqueName: \"kubernetes.io/projected/19640324-5142-4329-b8a4-63db6286cc9f-kube-api-access-499wj\") pod \"cert-manager-cainjector-559cdd8c87-mz47t\" (UID: \"19640324-5142-4329-b8a4-63db6286cc9f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.486902 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.486873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19640324-5142-4329-b8a4-63db6286cc9f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mz47t\" (UID: \"19640324-5142-4329-b8a4-63db6286cc9f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.487104 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.487081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-499wj\" (UniqueName: \"kubernetes.io/projected/19640324-5142-4329-b8a4-63db6286cc9f-kube-api-access-499wj\") pod \"cert-manager-cainjector-559cdd8c87-mz47t\" (UID: \"19640324-5142-4329-b8a4-63db6286cc9f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.557966 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.557891 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" Jul 20 22:58:19.684607 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:19.684582 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mz47t"] Jul 20 22:58:19.687048 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:58:19.687022 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19640324_5142_4329_b8a4_63db6286cc9f.slice/crio-8c9c87c6407952984b02acc9e0781e8c9c00b701181871a96170e51b87c5f7c2 WatchSource:0}: Error finding container 8c9c87c6407952984b02acc9e0781e8c9c00b701181871a96170e51b87c5f7c2: Status 404 returned error can't find the container with id 8c9c87c6407952984b02acc9e0781e8c9c00b701181871a96170e51b87c5f7c2 Jul 20 22:58:20.346259 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.346219 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" event={"ID":"19640324-5142-4329-b8a4-63db6286cc9f","Type":"ContainerStarted","Data":"8c9c87c6407952984b02acc9e0781e8c9c00b701181871a96170e51b87c5f7c2"} Jul 20 22:58:20.465853 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.465812 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:20.589929 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.589795 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptnsd\" (UniqueName: \"kubernetes.io/projected/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-kube-api-access-ptnsd\") pod \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " Jul 20 22:58:20.589929 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.589848 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-bundle\") pod \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " Jul 20 22:58:20.590630 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.589962 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-util\") pod \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\" (UID: \"061e9348-13bd-4876-a3ec-0d5ebbe3dedc\") " Jul 20 22:58:20.590630 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.590450 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-bundle" (OuterVolumeSpecName: "bundle") pod "061e9348-13bd-4876-a3ec-0d5ebbe3dedc" (UID: "061e9348-13bd-4876-a3ec-0d5ebbe3dedc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:58:20.592915 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.592885 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-kube-api-access-ptnsd" (OuterVolumeSpecName: "kube-api-access-ptnsd") pod "061e9348-13bd-4876-a3ec-0d5ebbe3dedc" (UID: "061e9348-13bd-4876-a3ec-0d5ebbe3dedc"). InnerVolumeSpecName "kube-api-access-ptnsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:58:20.596511 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.596429 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-util" (OuterVolumeSpecName: "util") pod "061e9348-13bd-4876-a3ec-0d5ebbe3dedc" (UID: "061e9348-13bd-4876-a3ec-0d5ebbe3dedc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:58:20.690894 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.690854 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-util\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:20.690894 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.690892 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptnsd\" (UniqueName: \"kubernetes.io/projected/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-kube-api-access-ptnsd\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:20.691154 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:20.690907 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061e9348-13bd-4876-a3ec-0d5ebbe3dedc-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:21.353547 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:21.353515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" event={"ID":"061e9348-13bd-4876-a3ec-0d5ebbe3dedc","Type":"ContainerDied","Data":"e0afbaf82618dc8e1812bfecb1f02001f12c1716093e1957e0c17785d25edf76"} Jul 20 22:58:21.353959 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:21.353571 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0afbaf82618dc8e1812bfecb1f02001f12c1716093e1957e0c17785d25edf76" Jul 20 22:58:21.353959 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:21.353664 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz5wlf" Jul 20 22:58:22.899766 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.899727 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4w5k9"] Jul 20 22:58:22.900161 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.900056 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerName="util" Jul 20 22:58:22.900161 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.900069 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerName="util" Jul 20 22:58:22.900161 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.900076 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerName="pull" Jul 20 22:58:22.900161 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.900081 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerName="pull" Jul 20 22:58:22.900161 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.900090 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerName="extract" Jul 20 22:58:22.900161 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.900097 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerName="extract" Jul 20 22:58:22.900161 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.900158 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="061e9348-13bd-4876-a3ec-0d5ebbe3dedc" containerName="extract" Jul 20 22:58:22.902970 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.902954 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:22.905760 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.905739 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-z5ccw\"" Jul 20 22:58:22.915365 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:22.915344 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4w5k9"] Jul 20 22:58:23.012305 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.012267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2g7d\" (UniqueName: \"kubernetes.io/projected/4ce9d182-73bf-4b4c-9582-47d61d115eef-kube-api-access-x2g7d\") pod \"cert-manager-55b4d954d5-4w5k9\" (UID: \"4ce9d182-73bf-4b4c-9582-47d61d115eef\") " pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:23.012468 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.012315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ce9d182-73bf-4b4c-9582-47d61d115eef-bound-sa-token\") pod \"cert-manager-55b4d954d5-4w5k9\" (UID: \"4ce9d182-73bf-4b4c-9582-47d61d115eef\") " pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:23.113067 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.113034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ce9d182-73bf-4b4c-9582-47d61d115eef-bound-sa-token\") pod \"cert-manager-55b4d954d5-4w5k9\" (UID: \"4ce9d182-73bf-4b4c-9582-47d61d115eef\") " pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:23.113209 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.113104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2g7d\" (UniqueName: \"kubernetes.io/projected/4ce9d182-73bf-4b4c-9582-47d61d115eef-kube-api-access-x2g7d\") pod \"cert-manager-55b4d954d5-4w5k9\" (UID: \"4ce9d182-73bf-4b4c-9582-47d61d115eef\") " pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:23.122993 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.122956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2g7d\" (UniqueName: \"kubernetes.io/projected/4ce9d182-73bf-4b4c-9582-47d61d115eef-kube-api-access-x2g7d\") pod \"cert-manager-55b4d954d5-4w5k9\" (UID: \"4ce9d182-73bf-4b4c-9582-47d61d115eef\") " pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:23.122993 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.122974 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4ce9d182-73bf-4b4c-9582-47d61d115eef-bound-sa-token\") pod \"cert-manager-55b4d954d5-4w5k9\" (UID: \"4ce9d182-73bf-4b4c-9582-47d61d115eef\") " pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:23.212107 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.212018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4w5k9" Jul 20 22:58:23.338318 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.338156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4w5k9"] Jul 20 22:58:23.340782 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:58:23.340758 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ce9d182_73bf_4b4c_9582_47d61d115eef.slice/crio-e4758257b482b8d9a625b2e1e1efb8838e3cf75e17b7c1f9caac31056fc57bd9 WatchSource:0}: Error finding container e4758257b482b8d9a625b2e1e1efb8838e3cf75e17b7c1f9caac31056fc57bd9: Status 404 returned error can't find the container with id e4758257b482b8d9a625b2e1e1efb8838e3cf75e17b7c1f9caac31056fc57bd9 Jul 20 22:58:23.363190 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.363153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4w5k9" event={"ID":"4ce9d182-73bf-4b4c-9582-47d61d115eef","Type":"ContainerStarted","Data":"e4758257b482b8d9a625b2e1e1efb8838e3cf75e17b7c1f9caac31056fc57bd9"} Jul 20 22:58:23.364537 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.364515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" event={"ID":"19640324-5142-4329-b8a4-63db6286cc9f","Type":"ContainerStarted","Data":"27a3bb1784e3f7cf40ee84cd432fdc61bb7db33f6ef29632eac3d78fec79348d"} Jul 20 22:58:23.380198 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:23.380156 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mz47t" podStartSLOduration=1.6564402569999999 podStartE2EDuration="4.380142354s" totalImagesPullingTime="2.723702097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:58:23.379644241 +0000 UTC m=+292.876098666" watchObservedRunningTime="2026-07-20 22:58:23.380142354 +0000 UTC m=+292.876596778" Jul 20 22:58:24.370332 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:24.370285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4w5k9" event={"ID":"4ce9d182-73bf-4b4c-9582-47d61d115eef","Type":"ContainerStarted","Data":"5ca21eae80f5e54a16c8f4060a9387bf9d0d451e07227eb4b6bae2ef66311b69"} Jul 20 22:58:24.393721 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:24.393667 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4w5k9" podStartSLOduration=2.393651169 podStartE2EDuration="2.393651169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:58:24.391602862 +0000 UTC m=+293.888057287" watchObservedRunningTime="2026-07-20 22:58:24.393651169 +0000 UTC m=+293.890105593" Jul 20 22:58:30.861829 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:30.861800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 22:58:30.863771 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:30.863744 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 22:58:30.867039 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:30.867019 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 22:58:30.869191 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:30.869165 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 22:58:30.871805 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:30.871784 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 22:58:41.997857 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:41.997819 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8"] Jul 20 22:58:42.001384 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.001362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.007524 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.003759 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s7wfd\"" Jul 20 22:58:42.007524 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.003998 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 20 22:58:42.007524 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.004641 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 20 22:58:42.011843 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.011807 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8"] Jul 20 22:58:42.181100 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.181056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.181277 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.181193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.181277 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.181240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9btb\" (UniqueName: \"kubernetes.io/projected/03031745-307e-44b2-9cff-d1f8fae5360e-kube-api-access-b9btb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.282147 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.282061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.282147 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.282112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9btb\" (UniqueName: \"kubernetes.io/projected/03031745-307e-44b2-9cff-d1f8fae5360e-kube-api-access-b9btb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.282147 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.282136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.282479 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.282457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.282479 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.282475 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.291415 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.291392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9btb\" (UniqueName: \"kubernetes.io/projected/03031745-307e-44b2-9cff-d1f8fae5360e-kube-api-access-b9btb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.315439 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.315415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:42.446521 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:42.446471 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8"] Jul 20 22:58:42.448472 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:58:42.448442 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03031745_307e_44b2_9cff_d1f8fae5360e.slice/crio-2a3db0b98d6be18891e0a071ed8149ff85b4c061c452e69dfea77fce6f4fe5da WatchSource:0}: Error finding container 2a3db0b98d6be18891e0a071ed8149ff85b4c061c452e69dfea77fce6f4fe5da: Status 404 returned error can't find the container with id 2a3db0b98d6be18891e0a071ed8149ff85b4c061c452e69dfea77fce6f4fe5da Jul 20 22:58:43.452853 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:43.452815 2574 generic.go:356] "Generic (PLEG): container finished" podID="03031745-307e-44b2-9cff-d1f8fae5360e" containerID="7e0145789dd2a76ff5df929b1902b2365200152301bcdb34777473965256ba5f" exitCode=0 Jul 20 22:58:43.453212 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:43.452906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" event={"ID":"03031745-307e-44b2-9cff-d1f8fae5360e","Type":"ContainerDied","Data":"7e0145789dd2a76ff5df929b1902b2365200152301bcdb34777473965256ba5f"} Jul 20 22:58:43.453212 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:43.452957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" event={"ID":"03031745-307e-44b2-9cff-d1f8fae5360e","Type":"ContainerStarted","Data":"2a3db0b98d6be18891e0a071ed8149ff85b4c061c452e69dfea77fce6f4fe5da"} Jul 20 22:58:43.453359 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:43.453344 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 22:58:44.459431 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:44.459399 2574 generic.go:356] "Generic (PLEG): container finished" podID="03031745-307e-44b2-9cff-d1f8fae5360e" containerID="ac953ee44ca3a1aa6a08d559a3033293734005f1ba1b25c20f6ca1ac69cd5195" exitCode=0 Jul 20 22:58:44.459875 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:44.459445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" event={"ID":"03031745-307e-44b2-9cff-d1f8fae5360e","Type":"ContainerDied","Data":"ac953ee44ca3a1aa6a08d559a3033293734005f1ba1b25c20f6ca1ac69cd5195"} Jul 20 22:58:45.466812 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:45.466781 2574 generic.go:356] "Generic (PLEG): container finished" podID="03031745-307e-44b2-9cff-d1f8fae5360e" containerID="82cb5935f0e96e5d097a5ba1d26303396694c50cbff562ff114c07c7967841c4" exitCode=0 Jul 20 22:58:45.467193 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:45.466854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" event={"ID":"03031745-307e-44b2-9cff-d1f8fae5360e","Type":"ContainerDied","Data":"82cb5935f0e96e5d097a5ba1d26303396694c50cbff562ff114c07c7967841c4"} Jul 20 22:58:46.594664 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.594639 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:46.719282 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.719248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-util\") pod \"03031745-307e-44b2-9cff-d1f8fae5360e\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " Jul 20 22:58:46.719282 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.719286 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-bundle\") pod \"03031745-307e-44b2-9cff-d1f8fae5360e\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " Jul 20 22:58:46.719517 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.719307 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9btb\" (UniqueName: \"kubernetes.io/projected/03031745-307e-44b2-9cff-d1f8fae5360e-kube-api-access-b9btb\") pod \"03031745-307e-44b2-9cff-d1f8fae5360e\" (UID: \"03031745-307e-44b2-9cff-d1f8fae5360e\") " Jul 20 22:58:46.720168 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.720140 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-bundle" (OuterVolumeSpecName: "bundle") pod "03031745-307e-44b2-9cff-d1f8fae5360e" (UID: "03031745-307e-44b2-9cff-d1f8fae5360e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:58:46.721380 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.721355 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03031745-307e-44b2-9cff-d1f8fae5360e-kube-api-access-b9btb" (OuterVolumeSpecName: "kube-api-access-b9btb") pod "03031745-307e-44b2-9cff-d1f8fae5360e" (UID: "03031745-307e-44b2-9cff-d1f8fae5360e"). InnerVolumeSpecName "kube-api-access-b9btb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 22:58:46.725389 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.725339 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-util" (OuterVolumeSpecName: "util") pod "03031745-307e-44b2-9cff-d1f8fae5360e" (UID: "03031745-307e-44b2-9cff-d1f8fae5360e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 22:58:46.820470 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.820434 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-util\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:46.820470 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.820463 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03031745-307e-44b2-9cff-d1f8fae5360e-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:46.820470 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:46.820474 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9btb\" (UniqueName: \"kubernetes.io/projected/03031745-307e-44b2-9cff-d1f8fae5360e-kube-api-access-b9btb\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 22:58:47.480795 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:47.480764 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" Jul 20 22:58:47.480795 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:47.480780 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq5s8" event={"ID":"03031745-307e-44b2-9cff-d1f8fae5360e","Type":"ContainerDied","Data":"2a3db0b98d6be18891e0a071ed8149ff85b4c061c452e69dfea77fce6f4fe5da"} Jul 20 22:58:47.481048 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:47.480852 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3db0b98d6be18891e0a071ed8149ff85b4c061c452e69dfea77fce6f4fe5da" Jul 20 22:58:54.086538 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086503 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt"] Jul 20 22:58:54.086936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086823 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03031745-307e-44b2-9cff-d1f8fae5360e" containerName="util" Jul 20 22:58:54.086936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086835 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03031745-307e-44b2-9cff-d1f8fae5360e" containerName="util" Jul 20 22:58:54.086936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086851 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03031745-307e-44b2-9cff-d1f8fae5360e" containerName="pull" Jul 20 22:58:54.086936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086859 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03031745-307e-44b2-9cff-d1f8fae5360e" containerName="pull" Jul 20 22:58:54.086936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086879 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03031745-307e-44b2-9cff-d1f8fae5360e" containerName="extract" Jul 20 22:58:54.086936 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086885 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03031745-307e-44b2-9cff-d1f8fae5360e" containerName="extract" Jul 20 22:58:54.087130 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.086955 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="03031745-307e-44b2-9cff-d1f8fae5360e" containerName="extract" Jul 20 22:58:54.089968 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.089950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.092772 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.092745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 22:58:54.093092 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.093059 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 22:58:54.093214 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.093095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nptdq\"" Jul 20 22:58:54.093214 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.093112 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 22:58:54.104233 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.104214 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 22:58:54.116607 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.116582 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt"] Jul 20 22:58:54.176168 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.176134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f125e432-a510-404a-9039-e45602ab288b-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.176331 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.176178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f125e432-a510-404a-9039-e45602ab288b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.176331 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.176273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqv4\" (UniqueName: \"kubernetes.io/projected/f125e432-a510-404a-9039-e45602ab288b-kube-api-access-9mqv4\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.176331 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.176312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f125e432-a510-404a-9039-e45602ab288b-tmp\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.176435 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.176370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f125e432-a510-404a-9039-e45602ab288b-manifests\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.276678 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.276645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f125e432-a510-404a-9039-e45602ab288b-manifests\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.276851 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.276687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f125e432-a510-404a-9039-e45602ab288b-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.276851 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.276716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f125e432-a510-404a-9039-e45602ab288b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.276851 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.276759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqv4\" (UniqueName: \"kubernetes.io/projected/f125e432-a510-404a-9039-e45602ab288b-kube-api-access-9mqv4\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.276851 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.276793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f125e432-a510-404a-9039-e45602ab288b-tmp\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.277063 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.277041 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f125e432-a510-404a-9039-e45602ab288b-manifests\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.279031 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.279012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f125e432-a510-404a-9039-e45602ab288b-tmp\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.279387 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.279366 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f125e432-a510-404a-9039-e45602ab288b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.279427 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.279374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f125e432-a510-404a-9039-e45602ab288b-webhook-cert\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.284657 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.284633 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqv4\" (UniqueName: \"kubernetes.io/projected/f125e432-a510-404a-9039-e45602ab288b-kube-api-access-9mqv4\") pod \"opendatahub-operator-controller-manager-699fd79659-ccxxt\" (UID: \"f125e432-a510-404a-9039-e45602ab288b\") " pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.400045 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.399992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:58:54.536983 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:54.536958 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt"] Jul 20 22:58:54.539258 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:58:54.539232 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf125e432_a510_404a_9039_e45602ab288b.slice/crio-9410db9d3fce83658096dc926770844f29697904f8246f6c798a68a1936fb572 WatchSource:0}: Error finding container 9410db9d3fce83658096dc926770844f29697904f8246f6c798a68a1936fb572: Status 404 returned error can't find the container with id 9410db9d3fce83658096dc926770844f29697904f8246f6c798a68a1936fb572 Jul 20 22:58:55.522009 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:55.521967 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" event={"ID":"f125e432-a510-404a-9039-e45602ab288b","Type":"ContainerStarted","Data":"9410db9d3fce83658096dc926770844f29697904f8246f6c798a68a1936fb572"} Jul 20 22:58:58.539772 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:58.539727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" event={"ID":"f125e432-a510-404a-9039-e45602ab288b","Type":"ContainerStarted","Data":"84d37c97ef055fdf07108565da5cf641926bb612ebf27f29a88967ccc7740ea8"} Jul 20 22:58:59.545746 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:59.545708 2574 generic.go:356] "Generic (PLEG): container finished" podID="f125e432-a510-404a-9039-e45602ab288b" containerID="84d37c97ef055fdf07108565da5cf641926bb612ebf27f29a88967ccc7740ea8" exitCode=0 Jul 20 22:58:59.546163 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:58:59.545762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" event={"ID":"f125e432-a510-404a-9039-e45602ab288b","Type":"ContainerDied","Data":"84d37c97ef055fdf07108565da5cf641926bb612ebf27f29a88967ccc7740ea8"} Jul 20 22:59:00.552205 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:00.552171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" event={"ID":"f125e432-a510-404a-9039-e45602ab288b","Type":"ContainerStarted","Data":"8ff44934c4fffda869f0d3b11be80dbebb14f5f51e775b3bc2def5c4c9996a1e"} Jul 20 22:59:00.552636 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:00.552310 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:59:00.574324 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:00.574274 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" podStartSLOduration=2.035593663 podStartE2EDuration="6.574257821s" totalImagesPullingTime="3.359231698s" totalInitContainerRuntime="1.17943246s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:58:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:00.572553336 +0000 UTC m=+330.069007760" watchObservedRunningTime="2026-07-20 22:59:00.574257821 +0000 UTC m=+330.070712244" Jul 20 22:59:11.558618 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:11.558522 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-699fd79659-ccxxt" Jul 20 22:59:13.735562 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.735520 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd"] Jul 20 22:59:13.744169 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.744140 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.751438 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.750966 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 22:59:13.752125 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.752099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 22:59:13.752125 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.752116 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 22:59:13.752329 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.752130 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7mfrk\"" Jul 20 22:59:13.752329 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.752099 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 22:59:13.759276 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.759257 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd"] Jul 20 22:59:13.845211 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.845178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpx9\" (UniqueName: \"kubernetes.io/projected/81a76f81-b28a-4956-abff-92d4a02a2a0c-kube-api-access-hvpx9\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.845211 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.845215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a76f81-b28a-4956-abff-92d4a02a2a0c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.845427 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.845243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81a76f81-b28a-4956-abff-92d4a02a2a0c-tmp\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.891374 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.891336 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vt2g2"] Jul 20 22:59:13.894875 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.894849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:13.897590 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.897293 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-p6fpm\"" Jul 20 22:59:13.897749 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.897728 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 20 22:59:13.905666 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.905638 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vt2g2"] Jul 20 22:59:13.945986 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.945952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpx9\" (UniqueName: \"kubernetes.io/projected/81a76f81-b28a-4956-abff-92d4a02a2a0c-kube-api-access-hvpx9\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.946177 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.945999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a76f81-b28a-4956-abff-92d4a02a2a0c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.946177 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.946043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxwv\" (UniqueName: \"kubernetes.io/projected/8cb9248c-af87-4f14-82f8-e2be59923b56-kube-api-access-mbxwv\") pod \"odh-model-controller-7998c67dc4-vt2g2\" (UID: \"8cb9248c-af87-4f14-82f8-e2be59923b56\") " pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:13.946177 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.946072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81a76f81-b28a-4956-abff-92d4a02a2a0c-tmp\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.946343 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:59:13.946194 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 20 22:59:13.946343 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:59:13.946259 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a76f81-b28a-4956-abff-92d4a02a2a0c-tls-certs podName:81a76f81-b28a-4956-abff-92d4a02a2a0c nodeName:}" failed. No retries permitted until 2026-07-20 22:59:14.446243911 +0000 UTC m=+343.942698316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/81a76f81-b28a-4956-abff-92d4a02a2a0c-tls-certs") pod "kube-auth-proxy-85d6bc8fc7-pnlcd" (UID: "81a76f81-b28a-4956-abff-92d4a02a2a0c") : secret "kube-auth-proxy-tls" not found Jul 20 22:59:13.946343 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.946298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9248c-af87-4f14-82f8-e2be59923b56-cert\") pod \"odh-model-controller-7998c67dc4-vt2g2\" (UID: \"8cb9248c-af87-4f14-82f8-e2be59923b56\") " pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:13.948453 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.948431 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81a76f81-b28a-4956-abff-92d4a02a2a0c-tmp\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:13.969158 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:13.969118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpx9\" (UniqueName: \"kubernetes.io/projected/81a76f81-b28a-4956-abff-92d4a02a2a0c-kube-api-access-hvpx9\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:14.047206 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.047104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxwv\" (UniqueName: \"kubernetes.io/projected/8cb9248c-af87-4f14-82f8-e2be59923b56-kube-api-access-mbxwv\") pod \"odh-model-controller-7998c67dc4-vt2g2\" (UID: \"8cb9248c-af87-4f14-82f8-e2be59923b56\") " pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:14.047206 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.047149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9248c-af87-4f14-82f8-e2be59923b56-cert\") pod \"odh-model-controller-7998c67dc4-vt2g2\" (UID: \"8cb9248c-af87-4f14-82f8-e2be59923b56\") " pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:14.047442 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:59:14.047291 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 20 22:59:14.047442 ip-10-0-134-37 kubenswrapper[2574]: E0720 22:59:14.047370 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb9248c-af87-4f14-82f8-e2be59923b56-cert podName:8cb9248c-af87-4f14-82f8-e2be59923b56 nodeName:}" failed. No retries permitted until 2026-07-20 22:59:14.547352261 +0000 UTC m=+344.043806664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8cb9248c-af87-4f14-82f8-e2be59923b56-cert") pod "odh-model-controller-7998c67dc4-vt2g2" (UID: "8cb9248c-af87-4f14-82f8-e2be59923b56") : secret "odh-model-controller-webhook-cert" not found Jul 20 22:59:14.071163 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.071124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxwv\" (UniqueName: \"kubernetes.io/projected/8cb9248c-af87-4f14-82f8-e2be59923b56-kube-api-access-mbxwv\") pod \"odh-model-controller-7998c67dc4-vt2g2\" (UID: \"8cb9248c-af87-4f14-82f8-e2be59923b56\") " pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:14.376357 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.376318 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2"] Jul 20 22:59:14.379927 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.379909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.382511 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.382460 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 20 22:59:14.382511 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.382461 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 22:59:14.382710 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.382461 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 22:59:14.382710 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.382461 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 20 22:59:14.382929 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.382915 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nnqf4\"" Jul 20 22:59:14.382969 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.382918 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 20 22:59:14.391050 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.391026 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2"] Jul 20 22:59:14.451210 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.451176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hwc\" (UniqueName: \"kubernetes.io/projected/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-kube-api-access-78hwc\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.451210 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.451213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-manager-config\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.451421 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.451237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-metrics-cert\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.451421 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.451277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a76f81-b28a-4956-abff-92d4a02a2a0c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:14.451421 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.451405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-cert\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.453637 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.453614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81a76f81-b28a-4956-abff-92d4a02a2a0c-tls-certs\") pod \"kube-auth-proxy-85d6bc8fc7-pnlcd\" (UID: \"81a76f81-b28a-4956-abff-92d4a02a2a0c\") " pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:14.551942 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.551900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-cert\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.552148 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.551958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hwc\" (UniqueName: \"kubernetes.io/projected/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-kube-api-access-78hwc\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.552148 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.551980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-manager-config\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.552148 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.552007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-metrics-cert\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.552148 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.552050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9248c-af87-4f14-82f8-e2be59923b56-cert\") pod \"odh-model-controller-7998c67dc4-vt2g2\" (UID: \"8cb9248c-af87-4f14-82f8-e2be59923b56\") " pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:14.552838 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.552815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-manager-config\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.554564 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.554530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-cert\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.554663 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.554572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-metrics-cert\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.554798 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.554782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cb9248c-af87-4f14-82f8-e2be59923b56-cert\") pod \"odh-model-controller-7998c67dc4-vt2g2\" (UID: \"8cb9248c-af87-4f14-82f8-e2be59923b56\") " pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:14.569567 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.569537 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hwc\" (UniqueName: \"kubernetes.io/projected/ff749a45-0f9d-41a0-9b8e-2e2e85225f69-kube-api-access-78hwc\") pod \"lws-controller-manager-87cc478df-p4pg2\" (UID: \"ff749a45-0f9d-41a0-9b8e-2e2e85225f69\") " pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.662566 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.662466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" Jul 20 22:59:14.689617 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.689583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:14.807794 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.807742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:14.841253 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.841226 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd"] Jul 20 22:59:14.844122 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:59:14.844088 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a76f81_b28a_4956_abff_92d4a02a2a0c.slice/crio-70b3dd15743b9f9cc6983509778bf1c2e72d25471a8b32d4bac6d3ee4c1e56ad WatchSource:0}: Error finding container 70b3dd15743b9f9cc6983509778bf1c2e72d25471a8b32d4bac6d3ee4c1e56ad: Status 404 returned error can't find the container with id 70b3dd15743b9f9cc6983509778bf1c2e72d25471a8b32d4bac6d3ee4c1e56ad Jul 20 22:59:14.868064 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.868029 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2"] Jul 20 22:59:14.871327 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:59:14.871287 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff749a45_0f9d_41a0_9b8e_2e2e85225f69.slice/crio-31789654cb464aaa545cafe35c667a1752b21ae2dcc6d67b7d579a28b13a118f WatchSource:0}: Error finding container 31789654cb464aaa545cafe35c667a1752b21ae2dcc6d67b7d579a28b13a118f: Status 404 returned error can't find the container with id 31789654cb464aaa545cafe35c667a1752b21ae2dcc6d67b7d579a28b13a118f Jul 20 22:59:14.958072 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:14.958048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-vt2g2"] Jul 20 22:59:14.961644 ip-10-0-134-37 kubenswrapper[2574]: W0720 22:59:14.961558 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb9248c_af87_4f14_82f8_e2be59923b56.slice/crio-a3a749c8e5cd4640bea8c447f0db7434606aa42571069da1c08e4fa92d0d8da8 WatchSource:0}: Error finding container a3a749c8e5cd4640bea8c447f0db7434606aa42571069da1c08e4fa92d0d8da8: Status 404 returned error can't find the container with id a3a749c8e5cd4640bea8c447f0db7434606aa42571069da1c08e4fa92d0d8da8 Jul 20 22:59:15.625509 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:15.625454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" event={"ID":"8cb9248c-af87-4f14-82f8-e2be59923b56","Type":"ContainerStarted","Data":"a3a749c8e5cd4640bea8c447f0db7434606aa42571069da1c08e4fa92d0d8da8"} Jul 20 22:59:15.627378 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:15.627341 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" event={"ID":"ff749a45-0f9d-41a0-9b8e-2e2e85225f69","Type":"ContainerStarted","Data":"31789654cb464aaa545cafe35c667a1752b21ae2dcc6d67b7d579a28b13a118f"} Jul 20 22:59:15.629070 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:15.629031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" event={"ID":"81a76f81-b28a-4956-abff-92d4a02a2a0c","Type":"ContainerStarted","Data":"70b3dd15743b9f9cc6983509778bf1c2e72d25471a8b32d4bac6d3ee4c1e56ad"} Jul 20 22:59:19.652694 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.652651 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" event={"ID":"8cb9248c-af87-4f14-82f8-e2be59923b56","Type":"ContainerStarted","Data":"d9288d4f1f6143024c8edc193dfdde24630562fb131f9f918afaca5e89e8f9d5"} Jul 20 22:59:19.653172 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.652768 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:19.654292 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.654266 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" event={"ID":"ff749a45-0f9d-41a0-9b8e-2e2e85225f69","Type":"ContainerStarted","Data":"dd8e6caa1c9532dd7df02d773d8b26b4469be73912900078a2a4e00adba5cbd9"} Jul 20 22:59:19.654447 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.654360 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 22:59:19.655845 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.655823 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" event={"ID":"81a76f81-b28a-4956-abff-92d4a02a2a0c","Type":"ContainerStarted","Data":"3a22dc64f1ad2c3d4581256da62b79cbce0a3672d3c999e54b0cd6725a90f467"} Jul 20 22:59:19.668172 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.668125 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" podStartSLOduration=3.010070252 podStartE2EDuration="6.668109904s" totalImagesPullingTime="3.658039652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:19.666749024 +0000 UTC m=+349.163203473" watchObservedRunningTime="2026-07-20 22:59:19.668109904 +0000 UTC m=+349.164564328" Jul 20 22:59:19.686848 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.686794 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85d6bc8fc7-pnlcd" podStartSLOduration=2.955662126 podStartE2EDuration="6.686777011s" totalImagesPullingTime="3.731114885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:19.686039892 +0000 UTC m=+349.182494307" watchObservedRunningTime="2026-07-20 22:59:19.686777011 +0000 UTC m=+349.183231446" Jul 20 22:59:19.700550 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:19.700471 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" podStartSLOduration=2.00276309 podStartE2EDuration="5.700443264s" totalImagesPullingTime="3.697680174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 22:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 22:59:19.69980657 +0000 UTC m=+349.196260994" watchObservedRunningTime="2026-07-20 22:59:19.700443264 +0000 UTC m=+349.196897687" Jul 20 22:59:30.662546 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:30.662446 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-vt2g2" Jul 20 22:59:30.663155 ip-10-0-134-37 kubenswrapper[2574]: I0720 22:59:30.662572 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-87cc478df-p4pg2" Jul 20 23:00:24.090430 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.090393 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5999c55d79-zdm9r"] Jul 20 23:00:24.093928 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.093902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.105170 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.105136 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5999c55d79-zdm9r"] Jul 20 23:00:24.222203 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.222160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-oauth-serving-cert\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.222203 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.222216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-config\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.222434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.222236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-service-ca\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.222434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.222310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-trusted-ca-bundle\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.222434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.222346 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-serving-cert\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.222434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.222375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-oauth-config\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.222434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.222416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbzzv\" (UniqueName: \"kubernetes.io/projected/dce6f6e6-d4e2-4d07-a5c1-678372281016-kube-api-access-dbzzv\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.323828 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.323779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-oauth-serving-cert\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324019 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.323841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-config\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324019 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.323870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-service-ca\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324019 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.323895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-trusted-ca-bundle\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324019 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.323916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-serving-cert\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324019 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.323955 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-oauth-config\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324302 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.324108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbzzv\" (UniqueName: \"kubernetes.io/projected/dce6f6e6-d4e2-4d07-a5c1-678372281016-kube-api-access-dbzzv\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324691 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.324665 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-service-ca\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324815 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.324699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-config\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324887 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.324837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-oauth-serving-cert\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.324946 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.324900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dce6f6e6-d4e2-4d07-a5c1-678372281016-trusted-ca-bundle\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.326616 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.326593 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-oauth-config\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.326616 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.326602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dce6f6e6-d4e2-4d07-a5c1-678372281016-console-serving-cert\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.332965 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.332942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbzzv\" (UniqueName: \"kubernetes.io/projected/dce6f6e6-d4e2-4d07-a5c1-678372281016-kube-api-access-dbzzv\") pod \"console-5999c55d79-zdm9r\" (UID: \"dce6f6e6-d4e2-4d07-a5c1-678372281016\") " pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.404929 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.404820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:24.548386 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.548356 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5999c55d79-zdm9r"] Jul 20 23:00:24.551189 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:00:24.551156 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce6f6e6_d4e2_4d07_a5c1_678372281016.slice/crio-a4965630738a5c5a9e2a6cccf7512ddb4d4e4d6383945e23d6c226c052c950e4 WatchSource:0}: Error finding container a4965630738a5c5a9e2a6cccf7512ddb4d4e4d6383945e23d6c226c052c950e4: Status 404 returned error can't find the container with id a4965630738a5c5a9e2a6cccf7512ddb4d4e4d6383945e23d6c226c052c950e4 Jul 20 23:00:24.959757 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.959718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5999c55d79-zdm9r" event={"ID":"dce6f6e6-d4e2-4d07-a5c1-678372281016","Type":"ContainerStarted","Data":"7a7d8fc75ed8b05cad2c2efec318b36ff51561a487786c6342bbfc5665f74e99"} Jul 20 23:00:24.959988 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.959769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5999c55d79-zdm9r" event={"ID":"dce6f6e6-d4e2-4d07-a5c1-678372281016","Type":"ContainerStarted","Data":"a4965630738a5c5a9e2a6cccf7512ddb4d4e4d6383945e23d6c226c052c950e4"} Jul 20 23:00:24.980682 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:24.980621 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5999c55d79-zdm9r" podStartSLOduration=0.98060279 podStartE2EDuration="980.60279ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:00:24.97961633 +0000 UTC m=+414.476070755" watchObservedRunningTime="2026-07-20 23:00:24.98060279 +0000 UTC m=+414.477057215" Jul 20 23:00:30.255396 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.255358 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph"] Jul 20 23:00:30.259117 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.259098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.261744 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.261717 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 23:00:30.261897 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.261803 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 23:00:30.262670 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.262650 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqlpd\"" Jul 20 23:00:30.267549 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.267520 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph"] Jul 20 23:00:30.379334 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.379295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.379334 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.379337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.379583 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.379435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhlw\" (UniqueName: \"kubernetes.io/projected/ba50946f-3dcf-456a-a79c-26e12a195f31-kube-api-access-5jhlw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.480627 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.480587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhlw\" (UniqueName: \"kubernetes.io/projected/ba50946f-3dcf-456a-a79c-26e12a195f31-kube-api-access-5jhlw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.480763 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.480653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.480763 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.480676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.481020 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.481005 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.481058 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.481045 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.489346 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.489319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhlw\" (UniqueName: \"kubernetes.io/projected/ba50946f-3dcf-456a-a79c-26e12a195f31-kube-api-access-5jhlw\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.570198 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.570109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:30.709640 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.709615 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph"] Jul 20 23:00:30.711521 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:00:30.711478 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba50946f_3dcf_456a_a79c_26e12a195f31.slice/crio-eaad54f56fad5119bcc779a58bd5712eda9ce194aacd5c2e513b9ec534fb81a5 WatchSource:0}: Error finding container eaad54f56fad5119bcc779a58bd5712eda9ce194aacd5c2e513b9ec534fb81a5: Status 404 returned error can't find the container with id eaad54f56fad5119bcc779a58bd5712eda9ce194aacd5c2e513b9ec534fb81a5 Jul 20 23:00:30.852514 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.852471 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk"] Jul 20 23:00:30.856047 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.856027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:30.863847 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.863826 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk"] Jul 20 23:00:30.984591 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.984554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:30.984792 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.984675 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:30.984792 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.984710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmp9q\" (UniqueName: \"kubernetes.io/projected/46361911-bbbf-4020-9faa-fb1855680140-kube-api-access-kmp9q\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:30.989848 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.989814 2574 generic.go:356] "Generic (PLEG): container finished" podID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerID="f1cbbf98dda3850cad3de15f2d8161c17bedae10b2c61efbfa9f7e8702246976" exitCode=0 Jul 20 23:00:30.989985 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.989883 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" event={"ID":"ba50946f-3dcf-456a-a79c-26e12a195f31","Type":"ContainerDied","Data":"f1cbbf98dda3850cad3de15f2d8161c17bedae10b2c61efbfa9f7e8702246976"} Jul 20 23:00:30.989985 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:30.989934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" event={"ID":"ba50946f-3dcf-456a-a79c-26e12a195f31","Type":"ContainerStarted","Data":"eaad54f56fad5119bcc779a58bd5712eda9ce194aacd5c2e513b9ec534fb81a5"} Jul 20 23:00:31.085203 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.085163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:31.085416 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.085245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmp9q\" (UniqueName: \"kubernetes.io/projected/46361911-bbbf-4020-9faa-fb1855680140-kube-api-access-kmp9q\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:31.085416 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.085342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:31.085656 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.085632 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:31.085716 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.085700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:31.094447 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.094427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmp9q\" (UniqueName: \"kubernetes.io/projected/46361911-bbbf-4020-9faa-fb1855680140-kube-api-access-kmp9q\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:31.192975 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.192892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:31.329861 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.329818 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk"] Jul 20 23:00:31.337044 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:00:31.337008 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46361911_bbbf_4020_9faa_fb1855680140.slice/crio-9d3cc188de5e9dae5f47185b7c4abed76fa61cb6848d8bb4b64e24ab03252974 WatchSource:0}: Error finding container 9d3cc188de5e9dae5f47185b7c4abed76fa61cb6848d8bb4b64e24ab03252974: Status 404 returned error can't find the container with id 9d3cc188de5e9dae5f47185b7c4abed76fa61cb6848d8bb4b64e24ab03252974 Jul 20 23:00:31.450608 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.450534 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55"] Jul 20 23:00:31.454017 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.454000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.461035 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.461007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55"] Jul 20 23:00:31.590074 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.590018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.590271 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.590132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktjg4\" (UniqueName: \"kubernetes.io/projected/169de9a2-173a-422f-82ae-4f5622517f68-kube-api-access-ktjg4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.590271 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.590157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.690914 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.690875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktjg4\" (UniqueName: \"kubernetes.io/projected/169de9a2-173a-422f-82ae-4f5622517f68-kube-api-access-ktjg4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.690914 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.690917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.691137 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.690973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.691365 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.691307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.691365 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.691337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.700278 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.700245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktjg4\" (UniqueName: \"kubernetes.io/projected/169de9a2-173a-422f-82ae-4f5622517f68-kube-api-access-ktjg4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.806909 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.806811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:31.852511 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.852466 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z"] Jul 20 23:00:31.856299 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.856272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:31.863456 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.863418 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z"] Jul 20 23:00:31.953806 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.953782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55"] Jul 20 23:00:31.961661 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:00:31.961630 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169de9a2_173a_422f_82ae_4f5622517f68.slice/crio-3a814e8b752443681519e6ec0c0c7083c0b6b6032a0e9725a599f15b450548e0 WatchSource:0}: Error finding container 3a814e8b752443681519e6ec0c0c7083c0b6b6032a0e9725a599f15b450548e0: Status 404 returned error can't find the container with id 3a814e8b752443681519e6ec0c0c7083c0b6b6032a0e9725a599f15b450548e0 Jul 20 23:00:31.996055 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.994940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:31.996055 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.994997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/90b470dc-df45-4d81-b1b3-b0b4b05c5156-kube-api-access-c92xp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:31.996055 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.995028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:31.998420 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.998389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" event={"ID":"169de9a2-173a-422f-82ae-4f5622517f68","Type":"ContainerStarted","Data":"3a814e8b752443681519e6ec0c0c7083c0b6b6032a0e9725a599f15b450548e0"} Jul 20 23:00:31.999645 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.999621 2574 generic.go:356] "Generic (PLEG): container finished" podID="46361911-bbbf-4020-9faa-fb1855680140" containerID="ae922756de89a2963bcade20640931bba67f3c41bfee0a96920b26d23ece985f" exitCode=0 Jul 20 23:00:31.999752 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.999652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" event={"ID":"46361911-bbbf-4020-9faa-fb1855680140","Type":"ContainerDied","Data":"ae922756de89a2963bcade20640931bba67f3c41bfee0a96920b26d23ece985f"} Jul 20 23:00:31.999752 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:31.999694 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" event={"ID":"46361911-bbbf-4020-9faa-fb1855680140","Type":"ContainerStarted","Data":"9d3cc188de5e9dae5f47185b7c4abed76fa61cb6848d8bb4b64e24ab03252974"} Jul 20 23:00:32.095536 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.095479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:32.095704 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.095561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/90b470dc-df45-4d81-b1b3-b0b4b05c5156-kube-api-access-c92xp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:32.095704 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.095594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:32.095941 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.095917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:32.095991 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.095948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:32.104737 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.104716 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/90b470dc-df45-4d81-b1b3-b0b4b05c5156-kube-api-access-c92xp\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:32.177528 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.177476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:32.308933 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:32.308875 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z"] Jul 20 23:00:32.310723 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:00:32.310697 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b470dc_df45_4d81_b1b3_b0b4b05c5156.slice/crio-714501bed72457e778cbd2de10bd8c975dfa1f1307c917ecbeb4afff2e2470f3 WatchSource:0}: Error finding container 714501bed72457e778cbd2de10bd8c975dfa1f1307c917ecbeb4afff2e2470f3: Status 404 returned error can't find the container with id 714501bed72457e778cbd2de10bd8c975dfa1f1307c917ecbeb4afff2e2470f3 Jul 20 23:00:33.006850 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.006815 2574 generic.go:356] "Generic (PLEG): container finished" podID="169de9a2-173a-422f-82ae-4f5622517f68" containerID="f37541c1ccd7b6ed1dade090b6549d2abff274b1334e28da097abdca23898c04" exitCode=0 Jul 20 23:00:33.009073 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.009045 2574 generic.go:356] "Generic (PLEG): container finished" podID="46361911-bbbf-4020-9faa-fb1855680140" containerID="377bba2e9f5ba2f1fe48a4ca0063dac7d8daf8d50dac3ccbd80addd90d670bea" exitCode=0 Jul 20 23:00:33.009200 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.009124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" event={"ID":"169de9a2-173a-422f-82ae-4f5622517f68","Type":"ContainerDied","Data":"f37541c1ccd7b6ed1dade090b6549d2abff274b1334e28da097abdca23898c04"} Jul 20 23:00:33.009200 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.009167 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" event={"ID":"46361911-bbbf-4020-9faa-fb1855680140","Type":"ContainerDied","Data":"377bba2e9f5ba2f1fe48a4ca0063dac7d8daf8d50dac3ccbd80addd90d670bea"} Jul 20 23:00:33.011275 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.011255 2574 generic.go:356] "Generic (PLEG): container finished" podID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerID="95cd1f0da5e4761ad9a840c3840e2190f9e968dd3315f8d4c9b3a324b90eb5e3" exitCode=0 Jul 20 23:00:33.011374 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.011339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" event={"ID":"ba50946f-3dcf-456a-a79c-26e12a195f31","Type":"ContainerDied","Data":"95cd1f0da5e4761ad9a840c3840e2190f9e968dd3315f8d4c9b3a324b90eb5e3"} Jul 20 23:00:33.012956 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.012935 2574 generic.go:356] "Generic (PLEG): container finished" podID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerID="4fac06687f2afc4711156167c0513d00ba5a98640c952bf4b9c8c3f3a61a75c3" exitCode=0 Jul 20 23:00:33.012956 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.012969 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" event={"ID":"90b470dc-df45-4d81-b1b3-b0b4b05c5156","Type":"ContainerDied","Data":"4fac06687f2afc4711156167c0513d00ba5a98640c952bf4b9c8c3f3a61a75c3"} Jul 20 23:00:33.013077 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:33.012996 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" event={"ID":"90b470dc-df45-4d81-b1b3-b0b4b05c5156","Type":"ContainerStarted","Data":"714501bed72457e778cbd2de10bd8c975dfa1f1307c917ecbeb4afff2e2470f3"} Jul 20 23:00:34.029818 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.029781 2574 generic.go:356] "Generic (PLEG): container finished" podID="46361911-bbbf-4020-9faa-fb1855680140" containerID="cff680103d32cfe226a21ad13e8cfe9982dd063355b1444580440e326e7ac2d4" exitCode=0 Jul 20 23:00:34.030252 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.029835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" event={"ID":"46361911-bbbf-4020-9faa-fb1855680140","Type":"ContainerDied","Data":"cff680103d32cfe226a21ad13e8cfe9982dd063355b1444580440e326e7ac2d4"} Jul 20 23:00:34.032078 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.032055 2574 generic.go:356] "Generic (PLEG): container finished" podID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerID="c0b09b5272880a560dcb16abb9b93b079b30f8b297067abb084538d28c1ddf98" exitCode=0 Jul 20 23:00:34.032218 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.032129 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" event={"ID":"ba50946f-3dcf-456a-a79c-26e12a195f31","Type":"ContainerDied","Data":"c0b09b5272880a560dcb16abb9b93b079b30f8b297067abb084538d28c1ddf98"} Jul 20 23:00:34.033981 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.033959 2574 generic.go:356] "Generic (PLEG): container finished" podID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerID="2eadd8d4ae53c37670e92a5cd9d5520235b45d1a91d804197ebefa0049eeebc9" exitCode=0 Jul 20 23:00:34.034077 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.034009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" event={"ID":"90b470dc-df45-4d81-b1b3-b0b4b05c5156","Type":"ContainerDied","Data":"2eadd8d4ae53c37670e92a5cd9d5520235b45d1a91d804197ebefa0049eeebc9"} Jul 20 23:00:34.405472 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.405439 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:34.405679 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.405521 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:34.410231 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:34.410208 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:35.041015 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.040978 2574 generic.go:356] "Generic (PLEG): container finished" podID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerID="2a2aa64165497c89fc3e9937bf701241754530c19e08183b62838cfd0a07a9ff" exitCode=0 Jul 20 23:00:35.041481 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.041027 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" event={"ID":"90b470dc-df45-4d81-b1b3-b0b4b05c5156","Type":"ContainerDied","Data":"2a2aa64165497c89fc3e9937bf701241754530c19e08183b62838cfd0a07a9ff"} Jul 20 23:00:35.042969 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.042943 2574 generic.go:356] "Generic (PLEG): container finished" podID="169de9a2-173a-422f-82ae-4f5622517f68" containerID="8c240d4143ed80ee0d45b24b5b26bdba5f1b952afa0c0f79ee6f82ccd50e8875" exitCode=0 Jul 20 23:00:35.043115 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.043051 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" event={"ID":"169de9a2-173a-422f-82ae-4f5622517f68","Type":"ContainerDied","Data":"8c240d4143ed80ee0d45b24b5b26bdba5f1b952afa0c0f79ee6f82ccd50e8875"} Jul 20 23:00:35.047591 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.047561 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5999c55d79-zdm9r" Jul 20 23:00:35.105346 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.105285 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c95987-hk2tx"] Jul 20 23:00:35.232252 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.232226 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:35.305748 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.305682 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:35.327735 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.327703 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-util\") pod \"ba50946f-3dcf-456a-a79c-26e12a195f31\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " Jul 20 23:00:35.327871 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.327772 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jhlw\" (UniqueName: \"kubernetes.io/projected/ba50946f-3dcf-456a-a79c-26e12a195f31-kube-api-access-5jhlw\") pod \"ba50946f-3dcf-456a-a79c-26e12a195f31\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " Jul 20 23:00:35.327871 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.327795 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-bundle\") pod \"ba50946f-3dcf-456a-a79c-26e12a195f31\" (UID: \"ba50946f-3dcf-456a-a79c-26e12a195f31\") " Jul 20 23:00:35.328361 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.328334 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-bundle" (OuterVolumeSpecName: "bundle") pod "ba50946f-3dcf-456a-a79c-26e12a195f31" (UID: "ba50946f-3dcf-456a-a79c-26e12a195f31"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:35.330051 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.330030 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba50946f-3dcf-456a-a79c-26e12a195f31-kube-api-access-5jhlw" (OuterVolumeSpecName: "kube-api-access-5jhlw") pod "ba50946f-3dcf-456a-a79c-26e12a195f31" (UID: "ba50946f-3dcf-456a-a79c-26e12a195f31"). InnerVolumeSpecName "kube-api-access-5jhlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:00:35.337591 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.337566 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-util" (OuterVolumeSpecName: "util") pod "ba50946f-3dcf-456a-a79c-26e12a195f31" (UID: "ba50946f-3dcf-456a-a79c-26e12a195f31"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:35.428345 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.428311 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmp9q\" (UniqueName: \"kubernetes.io/projected/46361911-bbbf-4020-9faa-fb1855680140-kube-api-access-kmp9q\") pod \"46361911-bbbf-4020-9faa-fb1855680140\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " Jul 20 23:00:35.428345 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.428349 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-util\") pod \"46361911-bbbf-4020-9faa-fb1855680140\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " Jul 20 23:00:35.428626 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.428381 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-bundle\") pod \"46361911-bbbf-4020-9faa-fb1855680140\" (UID: \"46361911-bbbf-4020-9faa-fb1855680140\") " Jul 20 23:00:35.428763 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.428743 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-util\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:35.428816 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.428769 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jhlw\" (UniqueName: \"kubernetes.io/projected/ba50946f-3dcf-456a-a79c-26e12a195f31-kube-api-access-5jhlw\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:35.428816 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.428780 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba50946f-3dcf-456a-a79c-26e12a195f31-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:35.428945 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.428922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-bundle" (OuterVolumeSpecName: "bundle") pod "46361911-bbbf-4020-9faa-fb1855680140" (UID: "46361911-bbbf-4020-9faa-fb1855680140"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:35.430440 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.430421 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46361911-bbbf-4020-9faa-fb1855680140-kube-api-access-kmp9q" (OuterVolumeSpecName: "kube-api-access-kmp9q") pod "46361911-bbbf-4020-9faa-fb1855680140" (UID: "46361911-bbbf-4020-9faa-fb1855680140"). InnerVolumeSpecName "kube-api-access-kmp9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:00:35.434316 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.434279 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-util" (OuterVolumeSpecName: "util") pod "46361911-bbbf-4020-9faa-fb1855680140" (UID: "46361911-bbbf-4020-9faa-fb1855680140"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:35.530094 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.530042 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:35.530094 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.530089 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmp9q\" (UniqueName: \"kubernetes.io/projected/46361911-bbbf-4020-9faa-fb1855680140-kube-api-access-kmp9q\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:35.530094 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:35.530099 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46361911-bbbf-4020-9faa-fb1855680140-util\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:36.049683 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.049646 2574 generic.go:356] "Generic (PLEG): container finished" podID="169de9a2-173a-422f-82ae-4f5622517f68" containerID="8a2b75cb1d0cf9b9fb99707331aa62b024fd7f645d91074e3aa43be52fba29de" exitCode=0 Jul 20 23:00:36.050114 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.049730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" event={"ID":"169de9a2-173a-422f-82ae-4f5622517f68","Type":"ContainerDied","Data":"8a2b75cb1d0cf9b9fb99707331aa62b024fd7f645d91074e3aa43be52fba29de"} Jul 20 23:00:36.051538 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.051521 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" Jul 20 23:00:36.051538 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.051527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk" event={"ID":"46361911-bbbf-4020-9faa-fb1855680140","Type":"ContainerDied","Data":"9d3cc188de5e9dae5f47185b7c4abed76fa61cb6848d8bb4b64e24ab03252974"} Jul 20 23:00:36.051660 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.051569 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d3cc188de5e9dae5f47185b7c4abed76fa61cb6848d8bb4b64e24ab03252974" Jul 20 23:00:36.053428 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.053405 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" Jul 20 23:00:36.053563 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.053437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph" event={"ID":"ba50946f-3dcf-456a-a79c-26e12a195f31","Type":"ContainerDied","Data":"eaad54f56fad5119bcc779a58bd5712eda9ce194aacd5c2e513b9ec534fb81a5"} Jul 20 23:00:36.053563 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.053478 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaad54f56fad5119bcc779a58bd5712eda9ce194aacd5c2e513b9ec534fb81a5" Jul 20 23:00:36.190233 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.190207 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:36.337143 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.337045 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-util\") pod \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " Jul 20 23:00:36.337143 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.337134 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-bundle\") pod \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " Jul 20 23:00:36.337355 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.337192 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/90b470dc-df45-4d81-b1b3-b0b4b05c5156-kube-api-access-c92xp\") pod \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\" (UID: \"90b470dc-df45-4d81-b1b3-b0b4b05c5156\") " Jul 20 23:00:36.337727 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.337696 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-bundle" (OuterVolumeSpecName: "bundle") pod "90b470dc-df45-4d81-b1b3-b0b4b05c5156" (UID: "90b470dc-df45-4d81-b1b3-b0b4b05c5156"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:36.339324 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.339303 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b470dc-df45-4d81-b1b3-b0b4b05c5156-kube-api-access-c92xp" (OuterVolumeSpecName: "kube-api-access-c92xp") pod "90b470dc-df45-4d81-b1b3-b0b4b05c5156" (UID: "90b470dc-df45-4d81-b1b3-b0b4b05c5156"). InnerVolumeSpecName "kube-api-access-c92xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:00:36.342981 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.342955 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-util" (OuterVolumeSpecName: "util") pod "90b470dc-df45-4d81-b1b3-b0b4b05c5156" (UID: "90b470dc-df45-4d81-b1b3-b0b4b05c5156"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:36.443240 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.438244 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92xp\" (UniqueName: \"kubernetes.io/projected/90b470dc-df45-4d81-b1b3-b0b4b05c5156-kube-api-access-c92xp\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:36.443585 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.443331 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-util\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:36.443585 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:36.443383 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90b470dc-df45-4d81-b1b3-b0b4b05c5156-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:37.060135 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.060080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" event={"ID":"90b470dc-df45-4d81-b1b3-b0b4b05c5156","Type":"ContainerDied","Data":"714501bed72457e778cbd2de10bd8c975dfa1f1307c917ecbeb4afff2e2470f3"} Jul 20 23:00:37.060648 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.060153 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="714501bed72457e778cbd2de10bd8c975dfa1f1307c917ecbeb4afff2e2470f3" Jul 20 23:00:37.060648 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.060114 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z" Jul 20 23:00:37.204163 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.204137 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:37.350714 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.350686 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-bundle\") pod \"169de9a2-173a-422f-82ae-4f5622517f68\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " Jul 20 23:00:37.350894 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.350759 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-util\") pod \"169de9a2-173a-422f-82ae-4f5622517f68\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " Jul 20 23:00:37.350894 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.350782 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktjg4\" (UniqueName: \"kubernetes.io/projected/169de9a2-173a-422f-82ae-4f5622517f68-kube-api-access-ktjg4\") pod \"169de9a2-173a-422f-82ae-4f5622517f68\" (UID: \"169de9a2-173a-422f-82ae-4f5622517f68\") " Jul 20 23:00:37.351288 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.351263 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-bundle" (OuterVolumeSpecName: "bundle") pod "169de9a2-173a-422f-82ae-4f5622517f68" (UID: "169de9a2-173a-422f-82ae-4f5622517f68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:37.352886 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.352863 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169de9a2-173a-422f-82ae-4f5622517f68-kube-api-access-ktjg4" (OuterVolumeSpecName: "kube-api-access-ktjg4") pod "169de9a2-173a-422f-82ae-4f5622517f68" (UID: "169de9a2-173a-422f-82ae-4f5622517f68"). InnerVolumeSpecName "kube-api-access-ktjg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:00:37.406719 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.406675 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-util" (OuterVolumeSpecName: "util") pod "169de9a2-173a-422f-82ae-4f5622517f68" (UID: "169de9a2-173a-422f-82ae-4f5622517f68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:00:37.452070 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.452033 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:37.452070 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.452064 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169de9a2-173a-422f-82ae-4f5622517f68-util\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:37.452241 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:37.452078 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktjg4\" (UniqueName: \"kubernetes.io/projected/169de9a2-173a-422f-82ae-4f5622517f68-kube-api-access-ktjg4\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:00:38.066697 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:38.066666 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" Jul 20 23:00:38.067108 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:38.066697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55" event={"ID":"169de9a2-173a-422f-82ae-4f5622517f68","Type":"ContainerDied","Data":"3a814e8b752443681519e6ec0c0c7083c0b6b6032a0e9725a599f15b450548e0"} Jul 20 23:00:38.067108 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:38.066741 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a814e8b752443681519e6ec0c0c7083c0b6b6032a0e9725a599f15b450548e0" Jul 20 23:00:55.987506 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987412 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859"] Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987745 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169de9a2-173a-422f-82ae-4f5622517f68" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987756 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="169de9a2-173a-422f-82ae-4f5622517f68" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987765 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46361911-bbbf-4020-9faa-fb1855680140" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987770 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="46361911-bbbf-4020-9faa-fb1855680140" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987779 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987784 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987794 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987799 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987804 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169de9a2-173a-422f-82ae-4f5622517f68" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987809 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="169de9a2-173a-422f-82ae-4f5622517f68" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987819 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169de9a2-173a-422f-82ae-4f5622517f68" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987824 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="169de9a2-173a-422f-82ae-4f5622517f68" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987829 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46361911-bbbf-4020-9faa-fb1855680140" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987834 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="46361911-bbbf-4020-9faa-fb1855680140" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987841 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987846 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987854 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987859 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987867 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987872 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerName="util" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987880 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46361911-bbbf-4020-9faa-fb1855680140" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987885 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="46361911-bbbf-4020-9faa-fb1855680140" containerName="pull" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987890 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987895 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987945 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b470dc-df45-4d81-b1b3-b0b4b05c5156" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987953 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="169de9a2-173a-422f-82ae-4f5622517f68" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987960 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="46361911-bbbf-4020-9faa-fb1855680140" containerName="extract" Jul 20 23:00:55.987955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.987968 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba50946f-3dcf-456a-a79c-26e12a195f31" containerName="extract" Jul 20 23:00:55.990907 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.990892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" Jul 20 23:00:55.993255 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.993233 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 20 23:00:55.993359 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.993284 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 23:00:55.994029 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.994013 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sv979\"" Jul 20 23:00:55.994106 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.994032 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 23:00:55.999509 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:55.999207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fps9c\" (UniqueName: \"kubernetes.io/projected/457b7481-e556-4d43-aa53-19c2da729772-kube-api-access-fps9c\") pod \"dns-operator-controller-manager-864dd78c4c-gh859\" (UID: \"457b7481-e556-4d43-aa53-19c2da729772\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" Jul 20 23:00:56.003707 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:56.003681 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859"] Jul 20 23:00:56.101314 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:56.101277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fps9c\" (UniqueName: \"kubernetes.io/projected/457b7481-e556-4d43-aa53-19c2da729772-kube-api-access-fps9c\") pod \"dns-operator-controller-manager-864dd78c4c-gh859\" (UID: \"457b7481-e556-4d43-aa53-19c2da729772\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" Jul 20 23:00:56.112286 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:56.112255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fps9c\" (UniqueName: \"kubernetes.io/projected/457b7481-e556-4d43-aa53-19c2da729772-kube-api-access-fps9c\") pod \"dns-operator-controller-manager-864dd78c4c-gh859\" (UID: \"457b7481-e556-4d43-aa53-19c2da729772\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" Jul 20 23:00:56.307269 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:56.307182 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" Jul 20 23:00:56.452358 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:56.452330 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859"] Jul 20 23:00:56.453808 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:00:56.453783 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457b7481_e556_4d43_aa53_19c2da729772.slice/crio-2b8a8626e24cf6c2baad9e7d2d07845e7c0b0453af307cd08bf6d9111c323edf WatchSource:0}: Error finding container 2b8a8626e24cf6c2baad9e7d2d07845e7c0b0453af307cd08bf6d9111c323edf: Status 404 returned error can't find the container with id 2b8a8626e24cf6c2baad9e7d2d07845e7c0b0453af307cd08bf6d9111c323edf Jul 20 23:00:57.160434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:57.160395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" event={"ID":"457b7481-e556-4d43-aa53-19c2da729772","Type":"ContainerStarted","Data":"2b8a8626e24cf6c2baad9e7d2d07845e7c0b0453af307cd08bf6d9111c323edf"} Jul 20 23:00:59.176592 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:59.176476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" event={"ID":"457b7481-e556-4d43-aa53-19c2da729772","Type":"ContainerStarted","Data":"e007f19b19b54dd33da6ab434f8dad3deb6b93a6a4e3c20d946aa5a6be4f5c92"} Jul 20 23:00:59.176592 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:59.176590 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" Jul 20 23:00:59.196459 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:00:59.196411 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" podStartSLOduration=1.735303818 podStartE2EDuration="4.196395898s" totalImagesPullingTime="2.46109208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:00:59.193308228 +0000 UTC m=+448.689762652" watchObservedRunningTime="2026-07-20 23:00:59.196395898 +0000 UTC m=+448.692850322" Jul 20 23:01:00.140695 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.140641 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68c95987-hk2tx" podUID="6d8e189e-e3a4-4a6a-afed-b4733564df78" containerName="console" containerID="cri-o://2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725" gracePeriod=15 Jul 20 23:01:00.402341 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.402279 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c95987-hk2tx" Jul 20 23:01:00.438546 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.438515 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-trusted-ca-bundle\") pod \"6d8e189e-e3a4-4a6a-afed-b4733564df78\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " Jul 20 23:01:00.438749 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.438559 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-serving-cert\") pod \"6d8e189e-e3a4-4a6a-afed-b4733564df78\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " Jul 20 23:01:00.438749 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.438603 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-oauth-serving-cert\") pod \"6d8e189e-e3a4-4a6a-afed-b4733564df78\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " Jul 20 23:01:00.438749 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.438626 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-config\") pod \"6d8e189e-e3a4-4a6a-afed-b4733564df78\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " Jul 20 23:01:00.438749 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.438663 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-oauth-config\") pod \"6d8e189e-e3a4-4a6a-afed-b4733564df78\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " Jul 20 23:01:00.438749 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.438713 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-service-ca\") pod \"6d8e189e-e3a4-4a6a-afed-b4733564df78\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " Jul 20 23:01:00.439019 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.438878 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmzdd\" (UniqueName: \"kubernetes.io/projected/6d8e189e-e3a4-4a6a-afed-b4733564df78-kube-api-access-fmzdd\") pod \"6d8e189e-e3a4-4a6a-afed-b4733564df78\" (UID: \"6d8e189e-e3a4-4a6a-afed-b4733564df78\") " Jul 20 23:01:00.439076 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.439036 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6d8e189e-e3a4-4a6a-afed-b4733564df78" (UID: "6d8e189e-e3a4-4a6a-afed-b4733564df78"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:01:00.439076 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.439041 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6d8e189e-e3a4-4a6a-afed-b4733564df78" (UID: "6d8e189e-e3a4-4a6a-afed-b4733564df78"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:01:00.439172 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.439102 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-service-ca" (OuterVolumeSpecName: "service-ca") pod "6d8e189e-e3a4-4a6a-afed-b4733564df78" (UID: "6d8e189e-e3a4-4a6a-afed-b4733564df78"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:01:00.439336 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.439309 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-oauth-serving-cert\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:00.439336 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.439337 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:00.439540 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.439351 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-trusted-ca-bundle\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:00.439540 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.439386 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-config" (OuterVolumeSpecName: "console-config") pod "6d8e189e-e3a4-4a6a-afed-b4733564df78" (UID: "6d8e189e-e3a4-4a6a-afed-b4733564df78"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:01:00.441109 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.441083 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6d8e189e-e3a4-4a6a-afed-b4733564df78" (UID: "6d8e189e-e3a4-4a6a-afed-b4733564df78"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:01:00.441109 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.441094 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8e189e-e3a4-4a6a-afed-b4733564df78-kube-api-access-fmzdd" (OuterVolumeSpecName: "kube-api-access-fmzdd") pod "6d8e189e-e3a4-4a6a-afed-b4733564df78" (UID: "6d8e189e-e3a4-4a6a-afed-b4733564df78"). InnerVolumeSpecName "kube-api-access-fmzdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:01:00.441231 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.441116 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6d8e189e-e3a4-4a6a-afed-b4733564df78" (UID: "6d8e189e-e3a4-4a6a-afed-b4733564df78"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:01:00.540475 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.540437 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-config\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:00.540475 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.540469 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-oauth-config\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:00.540475 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.540482 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzdd\" (UniqueName: \"kubernetes.io/projected/6d8e189e-e3a4-4a6a-afed-b4733564df78-kube-api-access-fmzdd\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:00.540732 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:00.540513 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d8e189e-e3a4-4a6a-afed-b4733564df78-console-serving-cert\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:01.187248 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.187213 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d8e189e-e3a4-4a6a-afed-b4733564df78" containerID="2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725" exitCode=0 Jul 20 23:01:01.187248 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.187252 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c95987-hk2tx" Jul 20 23:01:01.187442 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.187288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c95987-hk2tx" event={"ID":"6d8e189e-e3a4-4a6a-afed-b4733564df78","Type":"ContainerDied","Data":"2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725"} Jul 20 23:01:01.187442 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.187342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c95987-hk2tx" event={"ID":"6d8e189e-e3a4-4a6a-afed-b4733564df78","Type":"ContainerDied","Data":"4366b6d4ea6a5554ce6728272bbdecde10333b4d382d4fa04a75233f88e1e1a2"} Jul 20 23:01:01.187442 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.187365 2574 scope.go:117] "RemoveContainer" containerID="2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725" Jul 20 23:01:01.195958 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.195942 2574 scope.go:117] "RemoveContainer" containerID="2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725" Jul 20 23:01:01.196188 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:01:01.196171 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725\": container with ID starting with 2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725 not found: ID does not exist" containerID="2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725" Jul 20 23:01:01.196267 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.196196 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725"} err="failed to get container status \"2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725\": rpc error: code = NotFound desc = could not find container \"2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725\": container with ID starting with 2fd870b378d79c2477aece597020d3eba1bbabf361d520d0e00cdc3efdbb2725 not found: ID does not exist" Jul 20 23:01:01.211402 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.211361 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c95987-hk2tx"] Jul 20 23:01:01.213020 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:01.212998 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68c95987-hk2tx"] Jul 20 23:01:03.009299 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:03.009264 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8e189e-e3a4-4a6a-afed-b4733564df78" path="/var/lib/kubelet/pods/6d8e189e-e3a4-4a6a-afed-b4733564df78/volumes" Jul 20 23:01:10.183795 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:10.183765 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-gh859" Jul 20 23:01:47.820246 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.820209 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:01:47.820682 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.820535 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8e189e-e3a4-4a6a-afed-b4733564df78" containerName="console" Jul 20 23:01:47.820682 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.820546 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8e189e-e3a4-4a6a-afed-b4733564df78" containerName="console" Jul 20 23:01:47.820682 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.820619 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8e189e-e3a4-4a6a-afed-b4733564df78" containerName="console" Jul 20 23:01:47.826226 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.826210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:47.829047 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.829024 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 23:01:47.829442 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.829427 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqlpd\"" Jul 20 23:01:47.838274 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.838251 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:01:47.856506 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.856471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d94b952-f5bc-4653-b2b6-5b7add4dd7a7-config-file\") pod \"limitador-limitador-54d6d988d5-gnxpv\" (UID: \"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:47.856632 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.856575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6kjd\" (UniqueName: \"kubernetes.io/projected/2d94b952-f5bc-4653-b2b6-5b7add4dd7a7-kube-api-access-q6kjd\") pod \"limitador-limitador-54d6d988d5-gnxpv\" (UID: \"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:47.905097 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.905061 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:01:47.957168 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.957137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d94b952-f5bc-4653-b2b6-5b7add4dd7a7-config-file\") pod \"limitador-limitador-54d6d988d5-gnxpv\" (UID: \"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:47.957316 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.957198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6kjd\" (UniqueName: \"kubernetes.io/projected/2d94b952-f5bc-4653-b2b6-5b7add4dd7a7-kube-api-access-q6kjd\") pod \"limitador-limitador-54d6d988d5-gnxpv\" (UID: \"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:47.957777 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.957760 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d94b952-f5bc-4653-b2b6-5b7add4dd7a7-config-file\") pod \"limitador-limitador-54d6d988d5-gnxpv\" (UID: \"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:47.966240 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:47.966215 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6kjd\" (UniqueName: \"kubernetes.io/projected/2d94b952-f5bc-4653-b2b6-5b7add4dd7a7-kube-api-access-q6kjd\") pod \"limitador-limitador-54d6d988d5-gnxpv\" (UID: \"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:48.136196 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.136155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:48.184192 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.184156 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jl5nt"] Jul 20 23:01:48.188700 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.188548 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jl5nt" Jul 20 23:01:48.192692 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.192664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8qgsb\"" Jul 20 23:01:48.204897 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.204852 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jl5nt"] Jul 20 23:01:48.260460 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.260428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxt7\" (UniqueName: \"kubernetes.io/projected/63e6789e-dae2-4c22-8dfd-ac3d31f3e101-kube-api-access-lnxt7\") pod \"authorino-58cd955675-jl5nt\" (UID: \"63e6789e-dae2-4c22-8dfd-ac3d31f3e101\") " pod="kuadrant-system/authorino-58cd955675-jl5nt" Jul 20 23:01:48.277144 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.277119 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:01:48.278886 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:01:48.278859 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d94b952_f5bc_4653_b2b6_5b7add4dd7a7.slice/crio-e876b916a766c0b89d90792453d63c6a2a35579d052b9426f1f0b4fa52e229bf WatchSource:0}: Error finding container e876b916a766c0b89d90792453d63c6a2a35579d052b9426f1f0b4fa52e229bf: Status 404 returned error can't find the container with id e876b916a766c0b89d90792453d63c6a2a35579d052b9426f1f0b4fa52e229bf Jul 20 23:01:48.361841 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.361803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxt7\" (UniqueName: \"kubernetes.io/projected/63e6789e-dae2-4c22-8dfd-ac3d31f3e101-kube-api-access-lnxt7\") pod \"authorino-58cd955675-jl5nt\" (UID: \"63e6789e-dae2-4c22-8dfd-ac3d31f3e101\") " pod="kuadrant-system/authorino-58cd955675-jl5nt" Jul 20 23:01:48.374020 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.373990 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxt7\" (UniqueName: \"kubernetes.io/projected/63e6789e-dae2-4c22-8dfd-ac3d31f3e101-kube-api-access-lnxt7\") pod \"authorino-58cd955675-jl5nt\" (UID: \"63e6789e-dae2-4c22-8dfd-ac3d31f3e101\") " pod="kuadrant-system/authorino-58cd955675-jl5nt" Jul 20 23:01:48.432543 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.432433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" event={"ID":"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7","Type":"ContainerStarted","Data":"e876b916a766c0b89d90792453d63c6a2a35579d052b9426f1f0b4fa52e229bf"} Jul 20 23:01:48.502882 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.502849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jl5nt" Jul 20 23:01:48.635692 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:48.635654 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jl5nt"] Jul 20 23:01:48.637905 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:01:48.637877 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63e6789e_dae2_4c22_8dfd_ac3d31f3e101.slice/crio-346198ced64df23f75e540f1d97a9a51aeda31bae238c19ad2bf9582a6f78357 WatchSource:0}: Error finding container 346198ced64df23f75e540f1d97a9a51aeda31bae238c19ad2bf9582a6f78357: Status 404 returned error can't find the container with id 346198ced64df23f75e540f1d97a9a51aeda31bae238c19ad2bf9582a6f78357 Jul 20 23:01:49.442510 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:49.442447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jl5nt" event={"ID":"63e6789e-dae2-4c22-8dfd-ac3d31f3e101","Type":"ContainerStarted","Data":"346198ced64df23f75e540f1d97a9a51aeda31bae238c19ad2bf9582a6f78357"} Jul 20 23:01:53.374258 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:53.374224 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jl5nt"] Jul 20 23:01:53.471065 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:53.471026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" event={"ID":"2d94b952-f5bc-4653-b2b6-5b7add4dd7a7","Type":"ContainerStarted","Data":"d27b1f2420255167ea2f2d9e1484ca4e6d949b8bf173e6f62597e717c20c83de"} Jul 20 23:01:53.471247 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:53.471190 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:01:53.472430 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:53.472402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jl5nt" event={"ID":"63e6789e-dae2-4c22-8dfd-ac3d31f3e101","Type":"ContainerStarted","Data":"68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e"} Jul 20 23:01:53.487017 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:53.486971 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" podStartSLOduration=2.10202243 podStartE2EDuration="6.486957219s" totalImagesPullingTime="4.384934789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:53.485903276 +0000 UTC m=+502.982357700" watchObservedRunningTime="2026-07-20 23:01:53.486957219 +0000 UTC m=+502.983411684" Jul 20 23:01:53.499432 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:53.499389 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jl5nt" podStartSLOduration=1.466208285 podStartE2EDuration="5.499376768s" totalImagesPullingTime="4.033168483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:01:53.498158644 +0000 UTC m=+502.994613069" watchObservedRunningTime="2026-07-20 23:01:53.499376768 +0000 UTC m=+502.995831195" Jul 20 23:01:54.477503 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:54.477459 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jl5nt" podUID="63e6789e-dae2-4c22-8dfd-ac3d31f3e101" containerName="authorino" containerID="cri-o://68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e" gracePeriod=30 Jul 20 23:01:54.727214 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:54.727187 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jl5nt" Jul 20 23:01:54.821571 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:54.821459 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnxt7\" (UniqueName: \"kubernetes.io/projected/63e6789e-dae2-4c22-8dfd-ac3d31f3e101-kube-api-access-lnxt7\") pod \"63e6789e-dae2-4c22-8dfd-ac3d31f3e101\" (UID: \"63e6789e-dae2-4c22-8dfd-ac3d31f3e101\") " Jul 20 23:01:54.823519 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:54.823484 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e6789e-dae2-4c22-8dfd-ac3d31f3e101-kube-api-access-lnxt7" (OuterVolumeSpecName: "kube-api-access-lnxt7") pod "63e6789e-dae2-4c22-8dfd-ac3d31f3e101" (UID: "63e6789e-dae2-4c22-8dfd-ac3d31f3e101"). InnerVolumeSpecName "kube-api-access-lnxt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:01:54.923054 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:54.923002 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnxt7\" (UniqueName: \"kubernetes.io/projected/63e6789e-dae2-4c22-8dfd-ac3d31f3e101-kube-api-access-lnxt7\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:01:55.487136 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.487100 2574 generic.go:356] "Generic (PLEG): container finished" podID="63e6789e-dae2-4c22-8dfd-ac3d31f3e101" containerID="68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e" exitCode=0 Jul 20 23:01:55.487653 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.487151 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jl5nt" Jul 20 23:01:55.487653 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.487192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jl5nt" event={"ID":"63e6789e-dae2-4c22-8dfd-ac3d31f3e101","Type":"ContainerDied","Data":"68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e"} Jul 20 23:01:55.487653 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.487239 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jl5nt" event={"ID":"63e6789e-dae2-4c22-8dfd-ac3d31f3e101","Type":"ContainerDied","Data":"346198ced64df23f75e540f1d97a9a51aeda31bae238c19ad2bf9582a6f78357"} Jul 20 23:01:55.487653 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.487264 2574 scope.go:117] "RemoveContainer" containerID="68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e" Jul 20 23:01:55.495733 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.495716 2574 scope.go:117] "RemoveContainer" containerID="68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e" Jul 20 23:01:55.495981 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:01:55.495961 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e\": container with ID starting with 68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e not found: ID does not exist" containerID="68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e" Jul 20 23:01:55.496042 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.495993 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e"} err="failed to get container status \"68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e\": rpc error: code = NotFound desc = could not find container \"68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e\": container with ID starting with 68641c5426b982b75c2d459f4ab0f2c0caedf3a9c74b2245be5706083166206e not found: ID does not exist" Jul 20 23:01:55.502911 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.502887 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jl5nt"] Jul 20 23:01:55.506486 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:55.506466 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jl5nt"] Jul 20 23:01:57.009181 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:01:57.009141 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e6789e-dae2-4c22-8dfd-ac3d31f3e101" path="/var/lib/kubelet/pods/63e6789e-dae2-4c22-8dfd-ac3d31f3e101/volumes" Jul 20 23:02:04.478516 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:04.478457 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gnxpv" Jul 20 23:02:20.993628 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:20.993596 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g4jrv"] Jul 20 23:02:20.994053 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:20.993927 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e6789e-dae2-4c22-8dfd-ac3d31f3e101" containerName="authorino" Jul 20 23:02:20.994053 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:20.993937 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e6789e-dae2-4c22-8dfd-ac3d31f3e101" containerName="authorino" Jul 20 23:02:20.994053 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:20.993993 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e6789e-dae2-4c22-8dfd-ac3d31f3e101" containerName="authorino" Jul 20 23:02:20.998438 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:20.998415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g4jrv" Jul 20 23:02:21.000844 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.000814 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8qgsb\"" Jul 20 23:02:21.010089 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.010064 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g4jrv"] Jul 20 23:02:21.135796 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.135762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/573718cb-7153-4a12-bba9-6266ecea2ff0-kube-api-access-mc5jf\") pod \"authorino-5d4b67c446-g4jrv\" (UID: \"573718cb-7153-4a12-bba9-6266ecea2ff0\") " pod="kuadrant-system/authorino-5d4b67c446-g4jrv" Jul 20 23:02:21.227340 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.223669 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g4jrv"] Jul 20 23:02:21.227340 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:02:21.224229 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mc5jf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-g4jrv" podUID="573718cb-7153-4a12-bba9-6266ecea2ff0" Jul 20 23:02:21.236795 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.236768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/573718cb-7153-4a12-bba9-6266ecea2ff0-kube-api-access-mc5jf\") pod \"authorino-5d4b67c446-g4jrv\" (UID: \"573718cb-7153-4a12-bba9-6266ecea2ff0\") " pod="kuadrant-system/authorino-5d4b67c446-g4jrv" Jul 20 23:02:21.245089 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.245031 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/573718cb-7153-4a12-bba9-6266ecea2ff0-kube-api-access-mc5jf\") pod \"authorino-5d4b67c446-g4jrv\" (UID: \"573718cb-7153-4a12-bba9-6266ecea2ff0\") " pod="kuadrant-system/authorino-5d4b67c446-g4jrv" Jul 20 23:02:21.249839 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.249815 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78f646dbc4-6c4ls"] Jul 20 23:02:21.253257 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.253239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f646dbc4-6c4ls" Jul 20 23:02:21.258545 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.258508 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78f646dbc4-6c4ls"] Jul 20 23:02:21.338390 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.338355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvhs7\" (UniqueName: \"kubernetes.io/projected/dddf23e5-2bb1-4377-94fb-28f4de92bd31-kube-api-access-hvhs7\") pod \"authorino-78f646dbc4-6c4ls\" (UID: \"dddf23e5-2bb1-4377-94fb-28f4de92bd31\") " pod="kuadrant-system/authorino-78f646dbc4-6c4ls" Jul 20 23:02:21.372947 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.372916 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78f646dbc4-6c4ls"] Jul 20 23:02:21.373201 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:02:21.373180 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hvhs7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-78f646dbc4-6c4ls" podUID="dddf23e5-2bb1-4377-94fb-28f4de92bd31" Jul 20 23:02:21.400116 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.400085 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75b7656df4-gxlrw"] Jul 20 23:02:21.403425 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.403407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.405756 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.405736 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 20 23:02:21.411116 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.411093 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b7656df4-gxlrw"] Jul 20 23:02:21.439556 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.439529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvhs7\" (UniqueName: \"kubernetes.io/projected/dddf23e5-2bb1-4377-94fb-28f4de92bd31-kube-api-access-hvhs7\") pod \"authorino-78f646dbc4-6c4ls\" (UID: \"dddf23e5-2bb1-4377-94fb-28f4de92bd31\") " pod="kuadrant-system/authorino-78f646dbc4-6c4ls" Jul 20 23:02:21.447251 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.447224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvhs7\" (UniqueName: \"kubernetes.io/projected/dddf23e5-2bb1-4377-94fb-28f4de92bd31-kube-api-access-hvhs7\") pod \"authorino-78f646dbc4-6c4ls\" (UID: \"dddf23e5-2bb1-4377-94fb-28f4de92bd31\") " pod="kuadrant-system/authorino-78f646dbc4-6c4ls" Jul 20 23:02:21.540995 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.540909 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67e8d755-ce6d-41fe-b49b-c4e81364ea26-tls-cert\") pod \"authorino-75b7656df4-gxlrw\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.540995 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.540946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5h9z\" (UniqueName: \"kubernetes.io/projected/67e8d755-ce6d-41fe-b49b-c4e81364ea26-kube-api-access-s5h9z\") pod \"authorino-75b7656df4-gxlrw\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.619132 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.619099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f646dbc4-6c4ls" Jul 20 23:02:21.619266 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.619100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g4jrv" Jul 20 23:02:21.626203 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.626173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f646dbc4-6c4ls" Jul 20 23:02:21.629450 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.629430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g4jrv" Jul 20 23:02:21.641552 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.641532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67e8d755-ce6d-41fe-b49b-c4e81364ea26-tls-cert\") pod \"authorino-75b7656df4-gxlrw\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.641655 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.641556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5h9z\" (UniqueName: \"kubernetes.io/projected/67e8d755-ce6d-41fe-b49b-c4e81364ea26-kube-api-access-s5h9z\") pod \"authorino-75b7656df4-gxlrw\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.643935 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.643913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67e8d755-ce6d-41fe-b49b-c4e81364ea26-tls-cert\") pod \"authorino-75b7656df4-gxlrw\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.649291 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.649265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5h9z\" (UniqueName: \"kubernetes.io/projected/67e8d755-ce6d-41fe-b49b-c4e81364ea26-kube-api-access-s5h9z\") pod \"authorino-75b7656df4-gxlrw\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.712994 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.712947 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:02:21.741865 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.741839 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/573718cb-7153-4a12-bba9-6266ecea2ff0-kube-api-access-mc5jf\") pod \"573718cb-7153-4a12-bba9-6266ecea2ff0\" (UID: \"573718cb-7153-4a12-bba9-6266ecea2ff0\") " Jul 20 23:02:21.742054 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.741875 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvhs7\" (UniqueName: \"kubernetes.io/projected/dddf23e5-2bb1-4377-94fb-28f4de92bd31-kube-api-access-hvhs7\") pod \"dddf23e5-2bb1-4377-94fb-28f4de92bd31\" (UID: \"dddf23e5-2bb1-4377-94fb-28f4de92bd31\") " Jul 20 23:02:21.743948 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.743913 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573718cb-7153-4a12-bba9-6266ecea2ff0-kube-api-access-mc5jf" (OuterVolumeSpecName: "kube-api-access-mc5jf") pod "573718cb-7153-4a12-bba9-6266ecea2ff0" (UID: "573718cb-7153-4a12-bba9-6266ecea2ff0"). InnerVolumeSpecName "kube-api-access-mc5jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:02:21.744061 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.743924 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddf23e5-2bb1-4377-94fb-28f4de92bd31-kube-api-access-hvhs7" (OuterVolumeSpecName: "kube-api-access-hvhs7") pod "dddf23e5-2bb1-4377-94fb-28f4de92bd31" (UID: "dddf23e5-2bb1-4377-94fb-28f4de92bd31"). InnerVolumeSpecName "kube-api-access-hvhs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:02:21.842784 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.842748 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/573718cb-7153-4a12-bba9-6266ecea2ff0-kube-api-access-mc5jf\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:02:21.842784 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.842784 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvhs7\" (UniqueName: \"kubernetes.io/projected/dddf23e5-2bb1-4377-94fb-28f4de92bd31-kube-api-access-hvhs7\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:02:21.844913 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:21.844890 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b7656df4-gxlrw"] Jul 20 23:02:21.846765 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:02:21.846740 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e8d755_ce6d_41fe_b49b_c4e81364ea26.slice/crio-153c81493df4d515d453db3cb9631488329e3e07447fc415338ddd7e5d1f000a WatchSource:0}: Error finding container 153c81493df4d515d453db3cb9631488329e3e07447fc415338ddd7e5d1f000a: Status 404 returned error can't find the container with id 153c81493df4d515d453db3cb9631488329e3e07447fc415338ddd7e5d1f000a Jul 20 23:02:22.625926 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.625900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78f646dbc4-6c4ls" Jul 20 23:02:22.626310 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.626004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b7656df4-gxlrw" event={"ID":"67e8d755-ce6d-41fe-b49b-c4e81364ea26","Type":"ContainerStarted","Data":"81e36e666676ff0ff7d3e894308b134b88ad4a803a752165a0ba3d8c16ef7034"} Jul 20 23:02:22.626310 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.626040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b7656df4-gxlrw" event={"ID":"67e8d755-ce6d-41fe-b49b-c4e81364ea26","Type":"ContainerStarted","Data":"153c81493df4d515d453db3cb9631488329e3e07447fc415338ddd7e5d1f000a"} Jul 20 23:02:22.626310 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.626078 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g4jrv" Jul 20 23:02:22.641065 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.641025 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75b7656df4-gxlrw" podStartSLOduration=1.315924876 podStartE2EDuration="1.641009768s" totalImagesPullingTime="325.084892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:02:22.640316446 +0000 UTC m=+532.136770871" watchObservedRunningTime="2026-07-20 23:02:22.641009768 +0000 UTC m=+532.137464236" Jul 20 23:02:22.673067 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.673029 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g4jrv"] Jul 20 23:02:22.680781 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.680747 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g4jrv"] Jul 20 23:02:22.703258 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.703229 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78f646dbc4-6c4ls"] Jul 20 23:02:22.705183 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:22.705155 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78f646dbc4-6c4ls"] Jul 20 23:02:23.009107 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:23.008998 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573718cb-7153-4a12-bba9-6266ecea2ff0" path="/var/lib/kubelet/pods/573718cb-7153-4a12-bba9-6266ecea2ff0/volumes" Jul 20 23:02:23.009266 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:02:23.009252 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dddf23e5-2bb1-4377-94fb-28f4de92bd31" path="/var/lib/kubelet/pods/dddf23e5-2bb1-4377-94fb-28f4de92bd31/volumes" Jul 20 23:03:03.495499 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:03.495464 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:03:29.175955 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:29.175915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:03:30.896824 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:30.896793 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:03:30.898036 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:30.898015 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:03:30.901699 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:30.901681 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:03:30.903387 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:30.903366 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:03:41.867783 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:41.867745 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:03:50.708841 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.708801 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq"] Jul 20 23:03:50.712374 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.712356 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.714921 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.714897 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 23:03:50.715026 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.714897 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 23:03:50.715026 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.714947 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tmj2g\"" Jul 20 23:03:50.715652 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.715633 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 20 23:03:50.722099 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.722079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq"] Jul 20 23:03:50.796626 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.796585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.796792 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.796657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.796792 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.796686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.796792 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.796708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.796896 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.796803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad57446a-fac7-4c29-ac96-f3137a0d849e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.796896 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.796840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfhb\" (UniqueName: \"kubernetes.io/projected/ad57446a-fac7-4c29-ac96-f3137a0d849e-kube-api-access-8lfhb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.897623 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.897584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.897788 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.897647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.897788 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.897669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.897788 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.897692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.897788 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.897724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad57446a-fac7-4c29-ac96-f3137a0d849e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.897788 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.897747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfhb\" (UniqueName: \"kubernetes.io/projected/ad57446a-fac7-4c29-ac96-f3137a0d849e-kube-api-access-8lfhb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.898063 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.898036 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.898063 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.898054 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.898216 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.898124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.900030 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.900007 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad57446a-fac7-4c29-ac96-f3137a0d849e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.900341 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.900324 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad57446a-fac7-4c29-ac96-f3137a0d849e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:50.905252 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:50.905228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfhb\" (UniqueName: \"kubernetes.io/projected/ad57446a-fac7-4c29-ac96-f3137a0d849e-kube-api-access-8lfhb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rf4kq\" (UID: \"ad57446a-fac7-4c29-ac96-f3137a0d849e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:51.023550 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:51.023444 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:03:51.163165 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:51.163136 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq"] Jul 20 23:03:51.164625 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:03:51.164593 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad57446a_fac7_4c29_ac96_f3137a0d849e.slice/crio-c3b7b8ae6c4b5f7acbef4992b3ad46824e2c347d8a00770d7ebeb0b0683369c3 WatchSource:0}: Error finding container c3b7b8ae6c4b5f7acbef4992b3ad46824e2c347d8a00770d7ebeb0b0683369c3: Status 404 returned error can't find the container with id c3b7b8ae6c4b5f7acbef4992b3ad46824e2c347d8a00770d7ebeb0b0683369c3 Jul 20 23:03:51.166045 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:51.166022 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 23:03:51.779300 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:51.779267 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:03:52.081792 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:52.081701 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" event={"ID":"ad57446a-fac7-4c29-ac96-f3137a0d849e","Type":"ContainerStarted","Data":"c3b7b8ae6c4b5f7acbef4992b3ad46824e2c347d8a00770d7ebeb0b0683369c3"} Jul 20 23:03:53.088876 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:53.088791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" event={"ID":"ad57446a-fac7-4c29-ac96-f3137a0d849e","Type":"ContainerStarted","Data":"2482f6fbd0d7c6636dc151d08d6184066c692fc242d7cdfd8ed3f29b9b7caaa6"} Jul 20 23:03:53.106634 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:03:53.106575 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" podStartSLOduration=2.037255417 podStartE2EDuration="3.1065587s" totalImagesPullingTime="1.069303283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:03:53.105583745 +0000 UTC m=+622.602038172" watchObservedRunningTime="2026-07-20 23:03:53.1065587 +0000 UTC m=+622.603013115" Jul 20 23:04:01.023604 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:01.023571 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:04:01.024040 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:01.023821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:04:01.036152 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:01.036130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:04:01.141950 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:01.141922 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rf4kq" Jul 20 23:04:19.275815 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:19.275773 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:04:27.676098 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:27.676061 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:04:30.208732 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.208700 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx"] Jul 20 23:04:30.212656 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.212628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.215068 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.215048 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 20 23:04:30.222043 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.222006 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx"] Jul 20 23:04:30.260457 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.260437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.260632 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.260503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.260632 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.260526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86978c0c-7fb3-4b59-b07e-80dc29b110ce-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.260632 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.260551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74g8\" (UniqueName: \"kubernetes.io/projected/86978c0c-7fb3-4b59-b07e-80dc29b110ce-kube-api-access-j74g8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.260632 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.260572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.260632 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.260585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361226 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361226 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86978c0c-7fb3-4b59-b07e-80dc29b110ce-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361473 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74g8\" (UniqueName: \"kubernetes.io/projected/86978c0c-7fb3-4b59-b07e-80dc29b110ce-kube-api-access-j74g8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361473 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361473 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361473 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361701 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361608 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361831 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.361896 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.361851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.363602 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.363579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86978c0c-7fb3-4b59-b07e-80dc29b110ce-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.363861 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.363842 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86978c0c-7fb3-4b59-b07e-80dc29b110ce-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.369030 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.369003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74g8\" (UniqueName: \"kubernetes.io/projected/86978c0c-7fb3-4b59-b07e-80dc29b110ce-kube-api-access-j74g8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx\" (UID: \"86978c0c-7fb3-4b59-b07e-80dc29b110ce\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.524888 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.524802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:30.654936 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:30.654906 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx"] Jul 20 23:04:30.656675 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:04:30.656645 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86978c0c_7fb3_4b59_b07e_80dc29b110ce.slice/crio-2b33a7a43110913c3da7ae07c8abf47066569077df05426974808a4d67c69127 WatchSource:0}: Error finding container 2b33a7a43110913c3da7ae07c8abf47066569077df05426974808a4d67c69127: Status 404 returned error can't find the container with id 2b33a7a43110913c3da7ae07c8abf47066569077df05426974808a4d67c69127 Jul 20 23:04:31.286513 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:31.286463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" event={"ID":"86978c0c-7fb3-4b59-b07e-80dc29b110ce","Type":"ContainerStarted","Data":"c084bcce5a7a1acc723ab8fa20245cf611ebb78d81b63aee551148be12f8dc0f"} Jul 20 23:04:31.286916 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:31.286522 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" event={"ID":"86978c0c-7fb3-4b59-b07e-80dc29b110ce","Type":"ContainerStarted","Data":"2b33a7a43110913c3da7ae07c8abf47066569077df05426974808a4d67c69127"} Jul 20 23:04:31.305442 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:31.305394 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" podStartSLOduration=1.133672118 podStartE2EDuration="1.305380566s" totalImagesPullingTime="171.708448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:04:31.303615152 +0000 UTC m=+660.800069577" watchObservedRunningTime="2026-07-20 23:04:31.305380566 +0000 UTC m=+660.801834990" Jul 20 23:04:38.184570 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:38.184532 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:04:40.525680 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:40.525641 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:40.526070 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:40.525734 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:40.538027 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:40.538005 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:41.349404 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:41.349378 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx" Jul 20 23:04:54.270948 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.270906 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75b7656df4-gxlrw"] Jul 20 23:04:54.271593 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.271130 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75b7656df4-gxlrw" podUID="67e8d755-ce6d-41fe-b49b-c4e81364ea26" containerName="authorino" containerID="cri-o://81e36e666676ff0ff7d3e894308b134b88ad4a803a752165a0ba3d8c16ef7034" gracePeriod=30 Jul 20 23:04:54.407368 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.407338 2574 generic.go:356] "Generic (PLEG): container finished" podID="67e8d755-ce6d-41fe-b49b-c4e81364ea26" containerID="81e36e666676ff0ff7d3e894308b134b88ad4a803a752165a0ba3d8c16ef7034" exitCode=0 Jul 20 23:04:54.407563 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.407410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b7656df4-gxlrw" event={"ID":"67e8d755-ce6d-41fe-b49b-c4e81364ea26","Type":"ContainerDied","Data":"81e36e666676ff0ff7d3e894308b134b88ad4a803a752165a0ba3d8c16ef7034"} Jul 20 23:04:54.526410 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.526352 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:04:54.579880 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.579847 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67e8d755-ce6d-41fe-b49b-c4e81364ea26-tls-cert\") pod \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " Jul 20 23:04:54.580067 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.579971 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5h9z\" (UniqueName: \"kubernetes.io/projected/67e8d755-ce6d-41fe-b49b-c4e81364ea26-kube-api-access-s5h9z\") pod \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\" (UID: \"67e8d755-ce6d-41fe-b49b-c4e81364ea26\") " Jul 20 23:04:54.582781 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.582746 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e8d755-ce6d-41fe-b49b-c4e81364ea26-kube-api-access-s5h9z" (OuterVolumeSpecName: "kube-api-access-s5h9z") pod "67e8d755-ce6d-41fe-b49b-c4e81364ea26" (UID: "67e8d755-ce6d-41fe-b49b-c4e81364ea26"). InnerVolumeSpecName "kube-api-access-s5h9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:04:54.592099 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.592076 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e8d755-ce6d-41fe-b49b-c4e81364ea26-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "67e8d755-ce6d-41fe-b49b-c4e81364ea26" (UID: "67e8d755-ce6d-41fe-b49b-c4e81364ea26"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:04:54.680802 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.680760 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5h9z\" (UniqueName: \"kubernetes.io/projected/67e8d755-ce6d-41fe-b49b-c4e81364ea26-kube-api-access-s5h9z\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:04:54.680802 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:54.680792 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/67e8d755-ce6d-41fe-b49b-c4e81364ea26-tls-cert\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:04:55.413919 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:55.413888 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b7656df4-gxlrw" Jul 20 23:04:55.413919 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:55.413910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b7656df4-gxlrw" event={"ID":"67e8d755-ce6d-41fe-b49b-c4e81364ea26","Type":"ContainerDied","Data":"153c81493df4d515d453db3cb9631488329e3e07447fc415338ddd7e5d1f000a"} Jul 20 23:04:55.414412 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:55.413961 2574 scope.go:117] "RemoveContainer" containerID="81e36e666676ff0ff7d3e894308b134b88ad4a803a752165a0ba3d8c16ef7034" Jul 20 23:04:55.442976 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:55.442947 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75b7656df4-gxlrw"] Jul 20 23:04:55.450225 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:55.450199 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75b7656df4-gxlrw"] Jul 20 23:04:57.009035 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:04:57.008999 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e8d755-ce6d-41fe-b49b-c4e81364ea26" path="/var/lib/kubelet/pods/67e8d755-ce6d-41fe-b49b-c4e81364ea26/volumes" Jul 20 23:05:21.279931 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:05:21.279893 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:05:31.679631 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:05:31.679592 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:05:43.483627 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:05:43.483590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:07:25.281948 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:07:25.281913 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:07:37.281589 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:07:37.281552 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:08:08.278687 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:08.278648 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:08:18.271528 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:18.271479 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:08:26.478077 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:26.477996 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:08:30.935061 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:30.935034 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:08:30.936335 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:30.936311 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:08:30.940386 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:30.940361 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:08:30.941583 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:30.941555 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:08:35.976335 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:35.976299 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:08:46.971725 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:46.971688 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:08:55.877068 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:08:55.877029 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:09:07.374452 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:09:07.374415 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:09:17.373453 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:09:17.373419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:09:42.673816 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:09:42.673780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:09:54.366918 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:09:54.366838 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:10:16.474674 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:10:16.474640 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:10:28.667297 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:10:28.667263 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:10:42.384631 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:10:42.384590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:10:53.777362 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:10:53.777323 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:11:04.278680 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:11:04.278636 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:11:13.311130 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:11:13.311089 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:11:22.679740 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:11:22.679701 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:13:10.873898 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:10.873860 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:13:22.179807 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:22.179773 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:13:30.586171 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:30.586131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:13:30.967804 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:30.967775 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:13:30.970436 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:30.970416 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:13:30.973118 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:30.973095 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:13:30.977227 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:30.977118 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:13:41.981790 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:41.981749 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:13:51.871756 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:13:51.871718 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:14:01.169015 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:14:01.168982 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:14:09.180944 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:14:09.180908 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:14:18.768030 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:14:18.767993 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:14:26.984781 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:14:26.984690 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:14:37.476627 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:14:37.476584 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:14:48.666687 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:14:48.666651 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:14:54.868406 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:14:54.868366 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:15:00.139605 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.139569 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg"] Jul 20 23:15:00.140062 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.139947 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e8d755-ce6d-41fe-b49b-c4e81364ea26" containerName="authorino" Jul 20 23:15:00.140062 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.139958 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e8d755-ce6d-41fe-b49b-c4e81364ea26" containerName="authorino" Jul 20 23:15:00.140062 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.140022 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e8d755-ce6d-41fe-b49b-c4e81364ea26" containerName="authorino" Jul 20 23:15:00.143010 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.142993 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg" Jul 20 23:15:00.145360 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.145338 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:15:00.152825 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.152805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg" Jul 20 23:15:00.156824 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.156799 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg"] Jul 20 23:15:00.495552 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.495529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg"] Jul 20 23:15:00.497214 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:15:00.497178 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44d203b_5f19_4b2e_a6a1_4b2dca917483.slice/crio-cdd85728333d09f6ac116027a34fd5fc14d64a47bc31ba6b1be7644cedd59dd2 WatchSource:0}: Error finding container cdd85728333d09f6ac116027a34fd5fc14d64a47bc31ba6b1be7644cedd59dd2: Status 404 returned error can't find the container with id cdd85728333d09f6ac116027a34fd5fc14d64a47bc31ba6b1be7644cedd59dd2 Jul 20 23:15:00.499421 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:00.499402 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 23:15:01.478772 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:01.478731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg" event={"ID":"d44d203b-5f19-4b2e-a6a1-4b2dca917483","Type":"ContainerStarted","Data":"cdd85728333d09f6ac116027a34fd5fc14d64a47bc31ba6b1be7644cedd59dd2"} Jul 20 23:15:03.490801 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:03.490767 2574 generic.go:356] "Generic (PLEG): container finished" podID="d44d203b-5f19-4b2e-a6a1-4b2dca917483" containerID="665f8377e80afaed40bf89bbbe5862610919e573a4b0b1f47d297bf81d8573e9" exitCode=0 Jul 20 23:15:03.491186 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:03.490850 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg" event={"ID":"d44d203b-5f19-4b2e-a6a1-4b2dca917483","Type":"ContainerDied","Data":"665f8377e80afaed40bf89bbbe5862610919e573a4b0b1f47d297bf81d8573e9"} Jul 20 23:15:04.630311 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:04.630284 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg" Jul 20 23:15:05.504386 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:05.504358 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg" Jul 20 23:15:05.504596 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:05.504360 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743155-mhjsg" event={"ID":"d44d203b-5f19-4b2e-a6a1-4b2dca917483","Type":"ContainerDied","Data":"cdd85728333d09f6ac116027a34fd5fc14d64a47bc31ba6b1be7644cedd59dd2"} Jul 20 23:15:05.504596 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:05.504539 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd85728333d09f6ac116027a34fd5fc14d64a47bc31ba6b1be7644cedd59dd2" Jul 20 23:15:16.586817 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:16.586783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:15:26.074381 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:26.074342 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:15:37.878858 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:37.878819 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:15:49.473078 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:15:49.473045 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:16:17.575807 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:16:17.575725 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:16:31.625009 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:16:31.624974 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:16:39.180717 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:16:39.180675 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:16:46.881184 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:16:46.881143 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:16:54.881438 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:16:54.881400 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:17:04.269257 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:17:04.269221 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:17:12.880926 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:17:12.880887 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:17:20.977532 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:17:20.977482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:17:29.879900 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:17:29.879818 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:17:39.266901 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:17:39.266867 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:17:54.071360 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:17:54.071308 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:18:03.585314 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:03.585279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:18:13.284756 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:13.284724 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:18:22.675996 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:22.675961 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:18:31.002529 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:31.002474 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:18:31.008748 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:31.008723 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:18:31.009143 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:31.009125 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:18:31.013943 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:31.013925 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:18:34.972195 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:34.972157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:18:42.673182 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:42.673152 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:18:55.881471 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:18:55.881437 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:19:08.384183 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:19:08.384150 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:19:21.780044 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:19:21.779999 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:19:30.687245 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:19:30.687202 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:19:40.069822 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:19:40.069782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:19:47.275248 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:19:47.275205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:19:56.877534 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:19:56.877478 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:20:05.472150 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:05.472109 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:20:25.174740 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.174651 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc"] Jul 20 23:20:25.175305 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.175153 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44d203b-5f19-4b2e-a6a1-4b2dca917483" containerName="cleanup" Jul 20 23:20:25.175305 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.175171 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44d203b-5f19-4b2e-a6a1-4b2dca917483" containerName="cleanup" Jul 20 23:20:25.175305 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.175305 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44d203b-5f19-4b2e-a6a1-4b2dca917483" containerName="cleanup" Jul 20 23:20:25.178429 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.178407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" Jul 20 23:20:25.180702 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.180682 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:20:25.180781 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.180707 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:20:25.184791 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.184767 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc"] Jul 20 23:20:25.280016 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.279986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5beb22-3322-4aaa-8319-efe76a303c8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc\" (UID: \"9d5beb22-3322-4aaa-8319-efe76a303c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" Jul 20 23:20:25.380610 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.380575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5beb22-3322-4aaa-8319-efe76a303c8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc\" (UID: \"9d5beb22-3322-4aaa-8319-efe76a303c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" Jul 20 23:20:25.381155 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.381135 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5beb22-3322-4aaa-8319-efe76a303c8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc\" (UID: \"9d5beb22-3322-4aaa-8319-efe76a303c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" Jul 20 23:20:25.489703 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.489614 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" Jul 20 23:20:25.621517 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:25.621474 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc"] Jul 20 23:20:25.623134 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:20:25.623104 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5beb22_3322_4aaa_8319_efe76a303c8c.slice/crio-c0aa778926ae4172425dc7712b8879b9729580622bd47ea2b9318207ed6b6563 WatchSource:0}: Error finding container c0aa778926ae4172425dc7712b8879b9729580622bd47ea2b9318207ed6b6563: Status 404 returned error can't find the container with id c0aa778926ae4172425dc7712b8879b9729580622bd47ea2b9318207ed6b6563 Jul 20 23:20:26.171738 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:26.171649 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" event={"ID":"9d5beb22-3322-4aaa-8319-efe76a303c8c","Type":"ContainerStarted","Data":"ab6920b3667db72586c62cfe9ad52259b8bbcefbb9a3ff14fb3fb76c3138cd1b"} Jul 20 23:20:26.171738 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:26.171707 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" event={"ID":"9d5beb22-3322-4aaa-8319-efe76a303c8c","Type":"ContainerStarted","Data":"c0aa778926ae4172425dc7712b8879b9729580622bd47ea2b9318207ed6b6563"} Jul 20 23:20:26.188133 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:26.188089 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" podStartSLOduration=1.188076019 podStartE2EDuration="1.188076019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:20:26.186146899 +0000 UTC m=+1615.682601322" watchObservedRunningTime="2026-07-20 23:20:26.188076019 +0000 UTC m=+1615.684530443" Jul 20 23:20:27.177901 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:27.177864 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d5beb22-3322-4aaa-8319-efe76a303c8c" containerID="ab6920b3667db72586c62cfe9ad52259b8bbcefbb9a3ff14fb3fb76c3138cd1b" exitCode=0 Jul 20 23:20:27.178067 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:27.177908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" event={"ID":"9d5beb22-3322-4aaa-8319-efe76a303c8c","Type":"ContainerDied","Data":"ab6920b3667db72586c62cfe9ad52259b8bbcefbb9a3ff14fb3fb76c3138cd1b"} Jul 20 23:20:28.313952 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:28.313923 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" Jul 20 23:20:28.506081 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:28.506003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5beb22-3322-4aaa-8319-efe76a303c8c-maas-api-service-ca\") pod \"9d5beb22-3322-4aaa-8319-efe76a303c8c\" (UID: \"9d5beb22-3322-4aaa-8319-efe76a303c8c\") " Jul 20 23:20:28.506338 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:28.506314 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5beb22-3322-4aaa-8319-efe76a303c8c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d5beb22-3322-4aaa-8319-efe76a303c8c" (UID: "9d5beb22-3322-4aaa-8319-efe76a303c8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:20:28.607162 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:28.607123 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5beb22-3322-4aaa-8319-efe76a303c8c-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:20:29.188283 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:29.188255 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" Jul 20 23:20:29.188283 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:29.188263 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc" event={"ID":"9d5beb22-3322-4aaa-8319-efe76a303c8c","Type":"ContainerDied","Data":"c0aa778926ae4172425dc7712b8879b9729580622bd47ea2b9318207ed6b6563"} Jul 20 23:20:29.188471 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:29.188314 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0aa778926ae4172425dc7712b8879b9729580622bd47ea2b9318207ed6b6563" Jul 20 23:20:38.671436 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:38.671392 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:20:40.259096 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:40.259057 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc"] Jul 20 23:20:40.262937 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:40.262909 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e58bad50-8rhjc"] Jul 20 23:20:41.009773 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:41.009745 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5beb22-3322-4aaa-8319-efe76a303c8c" path="/var/lib/kubelet/pods/9d5beb22-3322-4aaa-8319-efe76a303c8c/volumes" Jul 20 23:20:49.473475 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:20:49.473436 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:21:08.682779 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.682743 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v"] Jul 20 23:21:08.683300 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.683251 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5beb22-3322-4aaa-8319-efe76a303c8c" containerName="revoke-keys" Jul 20 23:21:08.683300 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.683271 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5beb22-3322-4aaa-8319-efe76a303c8c" containerName="revoke-keys" Jul 20 23:21:08.683411 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.683394 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5beb22-3322-4aaa-8319-efe76a303c8c" containerName="revoke-keys" Jul 20 23:21:08.686398 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.686375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" Jul 20 23:21:08.688762 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.688736 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:21:08.688873 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.688821 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:21:08.693012 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.692990 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v"] Jul 20 23:21:08.754822 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.754785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d394741-3b1a-4f8e-b87b-ce68e8b1f285-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v\" (UID: \"6d394741-3b1a-4f8e-b87b-ce68e8b1f285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" Jul 20 23:21:08.855513 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.855461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d394741-3b1a-4f8e-b87b-ce68e8b1f285-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v\" (UID: \"6d394741-3b1a-4f8e-b87b-ce68e8b1f285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" Jul 20 23:21:08.856161 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.856134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d394741-3b1a-4f8e-b87b-ce68e8b1f285-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v\" (UID: \"6d394741-3b1a-4f8e-b87b-ce68e8b1f285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" Jul 20 23:21:08.997578 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:08.997467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" Jul 20 23:21:09.130341 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:09.130315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v"] Jul 20 23:21:09.132188 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:21:09.132159 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d394741_3b1a_4f8e_b87b_ce68e8b1f285.slice/crio-2461b2dbaffa0cb9613649022bdd7adcde3ccd5286e6507739e23e26e39f5d3e WatchSource:0}: Error finding container 2461b2dbaffa0cb9613649022bdd7adcde3ccd5286e6507739e23e26e39f5d3e: Status 404 returned error can't find the container with id 2461b2dbaffa0cb9613649022bdd7adcde3ccd5286e6507739e23e26e39f5d3e Jul 20 23:21:09.395292 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:09.395259 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" event={"ID":"6d394741-3b1a-4f8e-b87b-ce68e8b1f285","Type":"ContainerStarted","Data":"2461b2dbaffa0cb9613649022bdd7adcde3ccd5286e6507739e23e26e39f5d3e"} Jul 20 23:21:10.401768 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:10.401736 2574 generic.go:356] "Generic (PLEG): container finished" podID="6d394741-3b1a-4f8e-b87b-ce68e8b1f285" containerID="0e02a104135da17355ded53479988fe40323e9f07b15a9cf23e4040934e5c362" exitCode=0 Jul 20 23:21:10.402171 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:10.401816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" event={"ID":"6d394741-3b1a-4f8e-b87b-ce68e8b1f285","Type":"ContainerDied","Data":"0e02a104135da17355ded53479988fe40323e9f07b15a9cf23e4040934e5c362"} Jul 20 23:21:11.537059 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:11.537032 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" Jul 20 23:21:11.679658 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:11.679568 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d394741-3b1a-4f8e-b87b-ce68e8b1f285-maas-api-service-ca\") pod \"6d394741-3b1a-4f8e-b87b-ce68e8b1f285\" (UID: \"6d394741-3b1a-4f8e-b87b-ce68e8b1f285\") " Jul 20 23:21:11.679986 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:11.679954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d394741-3b1a-4f8e-b87b-ce68e8b1f285-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d394741-3b1a-4f8e-b87b-ce68e8b1f285" (UID: "6d394741-3b1a-4f8e-b87b-ce68e8b1f285"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:21:11.780307 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:11.780259 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d394741-3b1a-4f8e-b87b-ce68e8b1f285-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:21:12.415795 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:12.415768 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" Jul 20 23:21:12.415982 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:12.415801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v" event={"ID":"6d394741-3b1a-4f8e-b87b-ce68e8b1f285","Type":"ContainerDied","Data":"2461b2dbaffa0cb9613649022bdd7adcde3ccd5286e6507739e23e26e39f5d3e"} Jul 20 23:21:12.415982 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:12.415848 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2461b2dbaffa0cb9613649022bdd7adcde3ccd5286e6507739e23e26e39f5d3e" Jul 20 23:21:22.071959 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:22.071916 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:21:23.758161 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:23.758123 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v"] Jul 20 23:21:23.761177 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:23.761146 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5ae564de-l4j6v"] Jul 20 23:21:25.009388 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:25.009351 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d394741-3b1a-4f8e-b87b-ce68e8b1f285" path="/var/lib/kubelet/pods/6d394741-3b1a-4f8e-b87b-ce68e8b1f285/volumes" Jul 20 23:21:32.599000 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:32.598958 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:21:52.848412 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.848377 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44"] Jul 20 23:21:52.848909 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.848761 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d394741-3b1a-4f8e-b87b-ce68e8b1f285" containerName="revoke-keys" Jul 20 23:21:52.848909 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.848773 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d394741-3b1a-4f8e-b87b-ce68e8b1f285" containerName="revoke-keys" Jul 20 23:21:52.848909 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.848834 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d394741-3b1a-4f8e-b87b-ce68e8b1f285" containerName="revoke-keys" Jul 20 23:21:52.851923 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.851903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" Jul 20 23:21:52.854188 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.854164 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:21:52.854188 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.854180 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:21:52.858346 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.858320 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44"] Jul 20 23:21:52.927973 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:52.927943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d77d88-2ba3-4d57-93a4-b7ce8426d7a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-421a7b46-9hv44\" (UID: \"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" Jul 20 23:21:53.028836 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:53.028754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d77d88-2ba3-4d57-93a4-b7ce8426d7a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-421a7b46-9hv44\" (UID: \"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" Jul 20 23:21:53.029357 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:53.029337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d77d88-2ba3-4d57-93a4-b7ce8426d7a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-421a7b46-9hv44\" (UID: \"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" Jul 20 23:21:53.163712 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:53.163615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" Jul 20 23:21:53.308532 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:53.308480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44"] Jul 20 23:21:53.313300 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:21:53.313269 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d77d88_2ba3_4d57_93a4_b7ce8426d7a4.slice/crio-e25a5ac5e837f0b6f3656cabffc2746550f7e6924956fd7f91611dd5bf5a94de WatchSource:0}: Error finding container e25a5ac5e837f0b6f3656cabffc2746550f7e6924956fd7f91611dd5bf5a94de: Status 404 returned error can't find the container with id e25a5ac5e837f0b6f3656cabffc2746550f7e6924956fd7f91611dd5bf5a94de Jul 20 23:21:53.633977 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:53.633937 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" event={"ID":"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4","Type":"ContainerStarted","Data":"e25a5ac5e837f0b6f3656cabffc2746550f7e6924956fd7f91611dd5bf5a94de"} Jul 20 23:21:54.640980 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:54.640945 2574 generic.go:356] "Generic (PLEG): container finished" podID="58d77d88-2ba3-4d57-93a4-b7ce8426d7a4" containerID="2c3b07941575cd7b8e108c8cf40153a3a26707819a4f0d3101033b3fe122421b" exitCode=0 Jul 20 23:21:54.641374 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:54.641036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" event={"ID":"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4","Type":"ContainerDied","Data":"2c3b07941575cd7b8e108c8cf40153a3a26707819a4f0d3101033b3fe122421b"} Jul 20 23:21:55.779545 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:55.779515 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" Jul 20 23:21:55.857070 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:55.857034 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d77d88-2ba3-4d57-93a4-b7ce8426d7a4-maas-api-service-ca\") pod \"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4\" (UID: \"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4\") " Jul 20 23:21:55.857414 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:55.857389 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d77d88-2ba3-4d57-93a4-b7ce8426d7a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58d77d88-2ba3-4d57-93a4-b7ce8426d7a4" (UID: "58d77d88-2ba3-4d57-93a4-b7ce8426d7a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:21:55.957827 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:55.957736 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d77d88-2ba3-4d57-93a4-b7ce8426d7a4-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:21:56.652951 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:56.652906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" event={"ID":"58d77d88-2ba3-4d57-93a4-b7ce8426d7a4","Type":"ContainerDied","Data":"e25a5ac5e837f0b6f3656cabffc2746550f7e6924956fd7f91611dd5bf5a94de"} Jul 20 23:21:56.653139 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:56.652973 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e25a5ac5e837f0b6f3656cabffc2746550f7e6924956fd7f91611dd5bf5a94de" Jul 20 23:21:56.653139 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:21:56.652922 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44" Jul 20 23:22:06.075257 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:06.075217 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:22:07.920058 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:07.920013 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44"] Jul 20 23:22:07.922480 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:07.922452 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-421a7b46-9hv44"] Jul 20 23:22:09.008958 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:09.008923 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d77d88-2ba3-4d57-93a4-b7ce8426d7a4" path="/var/lib/kubelet/pods/58d77d88-2ba3-4d57-93a4-b7ce8426d7a4/volumes" Jul 20 23:22:15.074895 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:15.074853 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:22:36.902765 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.902726 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz"] Jul 20 23:22:36.903375 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.903257 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d77d88-2ba3-4d57-93a4-b7ce8426d7a4" containerName="revoke-keys" Jul 20 23:22:36.903375 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.903277 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d77d88-2ba3-4d57-93a4-b7ce8426d7a4" containerName="revoke-keys" Jul 20 23:22:36.903538 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.903380 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d77d88-2ba3-4d57-93a4-b7ce8426d7a4" containerName="revoke-keys" Jul 20 23:22:36.906421 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.906400 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" Jul 20 23:22:36.908756 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.908737 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:22:36.908860 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.908773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:22:36.914012 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.913712 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz"] Jul 20 23:22:36.998366 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:36.998330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed424bf8-0d11-4b35-a85e-2affc7a3efb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz\" (UID: \"ed424bf8-0d11-4b35-a85e-2affc7a3efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" Jul 20 23:22:37.099481 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:37.099440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed424bf8-0d11-4b35-a85e-2affc7a3efb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz\" (UID: \"ed424bf8-0d11-4b35-a85e-2affc7a3efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" Jul 20 23:22:37.100057 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:37.100036 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed424bf8-0d11-4b35-a85e-2affc7a3efb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz\" (UID: \"ed424bf8-0d11-4b35-a85e-2affc7a3efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" Jul 20 23:22:37.218459 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:37.218358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" Jul 20 23:22:37.348217 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:37.348193 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz"] Jul 20 23:22:37.350503 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:22:37.350467 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded424bf8_0d11_4b35_a85e_2affc7a3efb9.slice/crio-708af7ed57303af42aaaee4abefd79dbb969bc9c2950f15f0cd988159befb318 WatchSource:0}: Error finding container 708af7ed57303af42aaaee4abefd79dbb969bc9c2950f15f0cd988159befb318: Status 404 returned error can't find the container with id 708af7ed57303af42aaaee4abefd79dbb969bc9c2950f15f0cd988159befb318 Jul 20 23:22:37.874845 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:37.874814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" event={"ID":"ed424bf8-0d11-4b35-a85e-2affc7a3efb9","Type":"ContainerStarted","Data":"e40944ab6c33658c4d846690e2d266ab60245451fb697087e66fa4d49ada9356"} Jul 20 23:22:37.875008 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:37.874862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" event={"ID":"ed424bf8-0d11-4b35-a85e-2affc7a3efb9","Type":"ContainerStarted","Data":"708af7ed57303af42aaaee4abefd79dbb969bc9c2950f15f0cd988159befb318"} Jul 20 23:22:37.892251 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:37.892202 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" podStartSLOduration=1.892187998 podStartE2EDuration="1.892187998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:22:37.89016368 +0000 UTC m=+1747.386618104" watchObservedRunningTime="2026-07-20 23:22:37.892187998 +0000 UTC m=+1747.388642422" Jul 20 23:22:38.881724 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:38.881693 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed424bf8-0d11-4b35-a85e-2affc7a3efb9" containerID="e40944ab6c33658c4d846690e2d266ab60245451fb697087e66fa4d49ada9356" exitCode=0 Jul 20 23:22:38.882094 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:38.881756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" event={"ID":"ed424bf8-0d11-4b35-a85e-2affc7a3efb9","Type":"ContainerDied","Data":"e40944ab6c33658c4d846690e2d266ab60245451fb697087e66fa4d49ada9356"} Jul 20 23:22:40.017395 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:40.017366 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" Jul 20 23:22:40.120667 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:40.120635 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed424bf8-0d11-4b35-a85e-2affc7a3efb9-maas-api-service-ca\") pod \"ed424bf8-0d11-4b35-a85e-2affc7a3efb9\" (UID: \"ed424bf8-0d11-4b35-a85e-2affc7a3efb9\") " Jul 20 23:22:40.121000 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:40.120981 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed424bf8-0d11-4b35-a85e-2affc7a3efb9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed424bf8-0d11-4b35-a85e-2affc7a3efb9" (UID: "ed424bf8-0d11-4b35-a85e-2affc7a3efb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:22:40.221561 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:40.221462 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed424bf8-0d11-4b35-a85e-2affc7a3efb9-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:22:40.893723 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:40.893691 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" Jul 20 23:22:40.893723 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:40.893722 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz" event={"ID":"ed424bf8-0d11-4b35-a85e-2affc7a3efb9","Type":"ContainerDied","Data":"708af7ed57303af42aaaee4abefd79dbb969bc9c2950f15f0cd988159befb318"} Jul 20 23:22:40.893990 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:40.893765 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="708af7ed57303af42aaaee4abefd79dbb969bc9c2950f15f0cd988159befb318" Jul 20 23:22:50.472400 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:50.472360 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:22:51.972821 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:51.972783 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz"] Jul 20 23:22:51.974718 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:51.974696 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c1869ca5-9lczz"] Jul 20 23:22:53.008996 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:22:53.008958 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed424bf8-0d11-4b35-a85e-2affc7a3efb9" path="/var/lib/kubelet/pods/ed424bf8-0d11-4b35-a85e-2affc7a3efb9/volumes" Jul 20 23:23:31.039674 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:23:31.039643 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:23:31.045202 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:23:31.045176 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:23:31.048704 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:23:31.048682 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:23:31.053858 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:23:31.053839 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:24:16.472557 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:16.472518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:24:29.092115 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:29.092078 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:24:41.279021 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:41.278977 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:24:43.408840 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.408800 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt"] Jul 20 23:24:43.409242 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.409156 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed424bf8-0d11-4b35-a85e-2affc7a3efb9" containerName="revoke-keys" Jul 20 23:24:43.409242 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.409168 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed424bf8-0d11-4b35-a85e-2affc7a3efb9" containerName="revoke-keys" Jul 20 23:24:43.409242 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.409231 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed424bf8-0d11-4b35-a85e-2affc7a3efb9" containerName="revoke-keys" Jul 20 23:24:43.412283 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.412267 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" Jul 20 23:24:43.414590 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.414566 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:24:43.414759 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.414568 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:24:43.418984 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.418955 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt"] Jul 20 23:24:43.476458 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.476415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ab6b50-ca62-4bcc-9ca1-ae370de243bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt\" (UID: \"70ab6b50-ca62-4bcc-9ca1-ae370de243bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" Jul 20 23:24:43.577277 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.577246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ab6b50-ca62-4bcc-9ca1-ae370de243bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt\" (UID: \"70ab6b50-ca62-4bcc-9ca1-ae370de243bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" Jul 20 23:24:43.577873 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.577854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ab6b50-ca62-4bcc-9ca1-ae370de243bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt\" (UID: \"70ab6b50-ca62-4bcc-9ca1-ae370de243bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" Jul 20 23:24:43.724587 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:43.724474 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" Jul 20 23:24:44.069950 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:44.069923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt"] Jul 20 23:24:44.071991 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:24:44.071960 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ab6b50_ca62_4bcc_9ca1_ae370de243bb.slice/crio-416989c2a9845030bcf0ed60ab08ce2550cdcec9be6339b7246f361105242353 WatchSource:0}: Error finding container 416989c2a9845030bcf0ed60ab08ce2550cdcec9be6339b7246f361105242353: Status 404 returned error can't find the container with id 416989c2a9845030bcf0ed60ab08ce2550cdcec9be6339b7246f361105242353 Jul 20 23:24:44.560574 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:44.560529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" event={"ID":"70ab6b50-ca62-4bcc-9ca1-ae370de243bb","Type":"ContainerStarted","Data":"ee4b3f5fb28ce2de431ea5a1966783f760774c15065afe9c79cc0e185078497d"} Jul 20 23:24:44.561028 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:44.560589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" event={"ID":"70ab6b50-ca62-4bcc-9ca1-ae370de243bb","Type":"ContainerStarted","Data":"416989c2a9845030bcf0ed60ab08ce2550cdcec9be6339b7246f361105242353"} Jul 20 23:24:44.576518 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:44.576436 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" podStartSLOduration=1.576420622 podStartE2EDuration="1.576420622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:24:44.573800007 +0000 UTC m=+1874.070254431" watchObservedRunningTime="2026-07-20 23:24:44.576420622 +0000 UTC m=+1874.072875047" Jul 20 23:24:45.567884 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:45.567850 2574 generic.go:356] "Generic (PLEG): container finished" podID="70ab6b50-ca62-4bcc-9ca1-ae370de243bb" containerID="ee4b3f5fb28ce2de431ea5a1966783f760774c15065afe9c79cc0e185078497d" exitCode=0 Jul 20 23:24:45.568298 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:45.567934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" event={"ID":"70ab6b50-ca62-4bcc-9ca1-ae370de243bb","Type":"ContainerDied","Data":"ee4b3f5fb28ce2de431ea5a1966783f760774c15065afe9c79cc0e185078497d"} Jul 20 23:24:46.719211 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:46.719183 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" Jul 20 23:24:46.805289 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:46.805253 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ab6b50-ca62-4bcc-9ca1-ae370de243bb-maas-api-service-ca\") pod \"70ab6b50-ca62-4bcc-9ca1-ae370de243bb\" (UID: \"70ab6b50-ca62-4bcc-9ca1-ae370de243bb\") " Jul 20 23:24:46.805591 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:46.805568 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ab6b50-ca62-4bcc-9ca1-ae370de243bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70ab6b50-ca62-4bcc-9ca1-ae370de243bb" (UID: "70ab6b50-ca62-4bcc-9ca1-ae370de243bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:24:46.906622 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:46.906581 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ab6b50-ca62-4bcc-9ca1-ae370de243bb-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:24:47.580252 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:47.580223 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" Jul 20 23:24:47.580433 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:47.580269 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt" event={"ID":"70ab6b50-ca62-4bcc-9ca1-ae370de243bb","Type":"ContainerDied","Data":"416989c2a9845030bcf0ed60ab08ce2550cdcec9be6339b7246f361105242353"} Jul 20 23:24:47.580433 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:47.580315 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="416989c2a9845030bcf0ed60ab08ce2550cdcec9be6339b7246f361105242353" Jul 20 23:24:56.878541 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:56.878436 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:24:58.480433 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:58.480400 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt"] Jul 20 23:24:58.482420 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:58.482398 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e09171-jqdvt"] Jul 20 23:24:59.008923 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:24:59.008889 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ab6b50-ca62-4bcc-9ca1-ae370de243bb" path="/var/lib/kubelet/pods/70ab6b50-ca62-4bcc-9ca1-ae370de243bb/volumes" Jul 20 23:25:31.777095 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:31.777063 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:25:41.067606 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:41.067568 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:25:56.027807 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:56.027771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:25:59.337347 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.337306 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8"] Jul 20 23:25:59.337772 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.337663 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ab6b50-ca62-4bcc-9ca1-ae370de243bb" containerName="revoke-keys" Jul 20 23:25:59.337772 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.337674 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ab6b50-ca62-4bcc-9ca1-ae370de243bb" containerName="revoke-keys" Jul 20 23:25:59.337772 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.337748 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ab6b50-ca62-4bcc-9ca1-ae370de243bb" containerName="revoke-keys" Jul 20 23:25:59.340869 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.340851 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" Jul 20 23:25:59.343249 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.343221 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:25:59.343371 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.343266 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:25:59.348160 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.348132 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8"] Jul 20 23:25:59.452727 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.452690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911cd3e9-68ec-451a-b365-beebbe856d05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8\" (UID: \"911cd3e9-68ec-451a-b365-beebbe856d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" Jul 20 23:25:59.553444 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.553403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911cd3e9-68ec-451a-b365-beebbe856d05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8\" (UID: \"911cd3e9-68ec-451a-b365-beebbe856d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" Jul 20 23:25:59.554165 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.554138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911cd3e9-68ec-451a-b365-beebbe856d05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8\" (UID: \"911cd3e9-68ec-451a-b365-beebbe856d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" Jul 20 23:25:59.653453 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.653420 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" Jul 20 23:25:59.786769 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.786745 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8"] Jul 20 23:25:59.788791 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:25:59.788764 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911cd3e9_68ec_451a_b365_beebbe856d05.slice/crio-fd238e1896ecf4d6dee99c1bff0313f8aeca1012fc5049f9fea5b1c04808226a WatchSource:0}: Error finding container fd238e1896ecf4d6dee99c1bff0313f8aeca1012fc5049f9fea5b1c04808226a: Status 404 returned error can't find the container with id fd238e1896ecf4d6dee99c1bff0313f8aeca1012fc5049f9fea5b1c04808226a Jul 20 23:25:59.968829 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:25:59.968795 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" event={"ID":"911cd3e9-68ec-451a-b365-beebbe856d05","Type":"ContainerStarted","Data":"fd238e1896ecf4d6dee99c1bff0313f8aeca1012fc5049f9fea5b1c04808226a"} Jul 20 23:26:00.976420 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:00.976381 2574 generic.go:356] "Generic (PLEG): container finished" podID="911cd3e9-68ec-451a-b365-beebbe856d05" containerID="6a0c4717b995f4f3f579fa65f67ff725d9d9d30c6dc3bebf59944971eca1a484" exitCode=0 Jul 20 23:26:00.976843 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:00.976468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" event={"ID":"911cd3e9-68ec-451a-b365-beebbe856d05","Type":"ContainerDied","Data":"6a0c4717b995f4f3f579fa65f67ff725d9d9d30c6dc3bebf59944971eca1a484"} Jul 20 23:26:02.115906 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:02.115877 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" Jul 20 23:26:02.175923 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:02.175881 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911cd3e9-68ec-451a-b365-beebbe856d05-maas-api-service-ca\") pod \"911cd3e9-68ec-451a-b365-beebbe856d05\" (UID: \"911cd3e9-68ec-451a-b365-beebbe856d05\") " Jul 20 23:26:02.176300 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:02.176277 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911cd3e9-68ec-451a-b365-beebbe856d05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "911cd3e9-68ec-451a-b365-beebbe856d05" (UID: "911cd3e9-68ec-451a-b365-beebbe856d05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:26:02.277235 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:02.277132 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911cd3e9-68ec-451a-b365-beebbe856d05-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:26:02.994929 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:02.994890 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" event={"ID":"911cd3e9-68ec-451a-b365-beebbe856d05","Type":"ContainerDied","Data":"fd238e1896ecf4d6dee99c1bff0313f8aeca1012fc5049f9fea5b1c04808226a"} Jul 20 23:26:02.994929 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:02.994923 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8" Jul 20 23:26:02.995194 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:02.994967 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd238e1896ecf4d6dee99c1bff0313f8aeca1012fc5049f9fea5b1c04808226a" Jul 20 23:26:13.177307 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:13.177271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:26:19.417001 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:19.416960 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8"] Jul 20 23:26:19.419684 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:19.419659 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2db333cd-qjpv8"] Jul 20 23:26:21.010374 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:21.010340 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911cd3e9-68ec-451a-b365-beebbe856d05" path="/var/lib/kubelet/pods/911cd3e9-68ec-451a-b365-beebbe856d05/volumes" Jul 20 23:26:21.978666 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:21.978628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:26:31.111046 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:26:31.106696 2574 scope.go:117] "RemoveContainer" containerID="ab6920b3667db72586c62cfe9ad52259b8bbcefbb9a3ff14fb3fb76c3138cd1b" Jul 20 23:27:19.275043 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:27:19.274999 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:27:29.879620 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:27:29.879586 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:27:31.132945 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:27:31.132912 2574 scope.go:117] "RemoveContainer" containerID="0e02a104135da17355ded53479988fe40323e9f07b15a9cf23e4040934e5c362" Jul 20 23:28:19.279670 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:19.279586 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:28:31.075193 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:31.075164 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:28:31.081024 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:31.081001 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:28:31.085208 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:31.085187 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:28:31.090324 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:31.090307 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:28:31.156038 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:31.156005 2574 scope.go:117] "RemoveContainer" containerID="2c3b07941575cd7b8e108c8cf40153a3a26707819a4f0d3101033b3fe122421b" Jul 20 23:28:33.173100 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:33.173063 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:28:55.077890 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:28:55.077849 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:29:23.820159 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.820085 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2"] Jul 20 23:29:23.822441 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.820417 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911cd3e9-68ec-451a-b365-beebbe856d05" containerName="revoke-keys" Jul 20 23:29:23.822441 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.820427 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="911cd3e9-68ec-451a-b365-beebbe856d05" containerName="revoke-keys" Jul 20 23:29:23.822441 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.820516 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="911cd3e9-68ec-451a-b365-beebbe856d05" containerName="revoke-keys" Jul 20 23:29:23.823472 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.823456 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.827008 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.826987 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-324c54\"/\"test-model-324c54-kserve-self-signed-certs\"" Jul 20 23:29:23.827131 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.827004 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-324c54\"/\"openshift-service-ca.crt\"" Jul 20 23:29:23.827131 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.827019 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-324c54\"/\"default-dockercfg-smczp\"" Jul 20 23:29:23.827131 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.826991 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-324c54\"/\"kube-root-ca.crt\"" Jul 20 23:29:23.831058 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.831041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-dshm\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.831168 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.831081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-model-cache\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.831168 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.831103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4hw\" (UniqueName: \"kubernetes.io/projected/1bc78569-94cb-49c0-94fc-eb1d73356aac-kube-api-access-md4hw\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.831168 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.831155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc78569-94cb-49c0-94fc-eb1d73356aac-tls-certs\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.831324 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.831187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-home\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.831324 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.831238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-tmp-dir\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.831883 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.831861 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2"] Jul 20 23:29:23.931744 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.931715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-model-cache\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.931898 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.931753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4hw\" (UniqueName: \"kubernetes.io/projected/1bc78569-94cb-49c0-94fc-eb1d73356aac-kube-api-access-md4hw\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.931898 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.931803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc78569-94cb-49c0-94fc-eb1d73356aac-tls-certs\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.931898 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.931836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-home\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.931898 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.931857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-tmp-dir\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.932109 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.931939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-dshm\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.932216 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.932191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-model-cache\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.932287 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.932224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-home\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.932287 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.932267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-tmp-dir\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.934147 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.934119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-dshm\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.934270 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.934249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc78569-94cb-49c0-94fc-eb1d73356aac-tls-certs\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:23.939587 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:23.939565 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4hw\" (UniqueName: \"kubernetes.io/projected/1bc78569-94cb-49c0-94fc-eb1d73356aac-kube-api-access-md4hw\") pod \"test-model-324c54-kserve-68b9bdff5f-qf2h2\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:24.132640 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:24.132609 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:24.266576 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:24.266547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2"] Jul 20 23:29:24.268390 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:29:24.268366 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc78569_94cb_49c0_94fc_eb1d73356aac.slice/crio-366c1bac460e3740fa93736db5f2f9381cf8cfe9f37ebd8fe9077bdf72916b33 WatchSource:0}: Error finding container 366c1bac460e3740fa93736db5f2f9381cf8cfe9f37ebd8fe9077bdf72916b33: Status 404 returned error can't find the container with id 366c1bac460e3740fa93736db5f2f9381cf8cfe9f37ebd8fe9077bdf72916b33 Jul 20 23:29:25.076850 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:25.076819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" event={"ID":"1bc78569-94cb-49c0-94fc-eb1d73356aac","Type":"ContainerStarted","Data":"dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03"} Jul 20 23:29:25.076850 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:25.076857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" event={"ID":"1bc78569-94cb-49c0-94fc-eb1d73356aac","Type":"ContainerStarted","Data":"366c1bac460e3740fa93736db5f2f9381cf8cfe9f37ebd8fe9077bdf72916b33"} Jul 20 23:29:25.092412 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:25.092357 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" podStartSLOduration=2.092342014 podStartE2EDuration="2.092342014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:29:25.091550062 +0000 UTC m=+2154.588004485" watchObservedRunningTime="2026-07-20 23:29:25.092342014 +0000 UTC m=+2154.588796438" Jul 20 23:29:28.078106 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:28.078067 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:29:31.176460 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:31.176433 2574 scope.go:117] "RemoveContainer" containerID="e40944ab6c33658c4d846690e2d266ab60245451fb697087e66fa4d49ada9356" Jul 20 23:29:34.133666 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:34.133630 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:34.133666 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:34.133667 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:34.146079 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:34.146048 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:35.138062 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:35.138027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:29:41.670000 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:29:41.669965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:30:00.136660 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.136622 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6"] Jul 20 23:30:00.140143 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.140125 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv"] Jul 20 23:30:00.140287 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.140265 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" Jul 20 23:30:00.142678 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.142661 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:30:00.143344 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.143330 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt"] Jul 20 23:30:00.143454 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.143441 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" Jul 20 23:30:00.147840 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.147818 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6"] Jul 20 23:30:00.148070 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.148056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" Jul 20 23:30:00.151345 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.151324 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv"] Jul 20 23:30:00.154696 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.154667 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt"] Jul 20 23:30:00.155362 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.155343 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" Jul 20 23:30:00.164362 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.164344 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" Jul 20 23:30:00.202477 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.202452 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" Jul 20 23:30:00.326978 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.326813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv"] Jul 20 23:30:00.330024 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:30:00.329998 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22909265_657b_4701_a98c_cf6c6e52e605.slice/crio-201c26d068f8b6b2ce8fa4a83d773710f2325ce08a96d3ea87ec2cae4a15a67c WatchSource:0}: Error finding container 201c26d068f8b6b2ce8fa4a83d773710f2325ce08a96d3ea87ec2cae4a15a67c: Status 404 returned error can't find the container with id 201c26d068f8b6b2ce8fa4a83d773710f2325ce08a96d3ea87ec2cae4a15a67c Jul 20 23:30:00.371699 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.371672 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt"] Jul 20 23:30:00.372903 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:30:00.372879 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a29b6a3_9213_4fcf_b721_a10f10822d94.slice/crio-8498a243f18b6b10d9d2219cbc86f4db5ccfa8a47d71b0151c4642c2a14269df WatchSource:0}: Error finding container 8498a243f18b6b10d9d2219cbc86f4db5ccfa8a47d71b0151c4642c2a14269df: Status 404 returned error can't find the container with id 8498a243f18b6b10d9d2219cbc86f4db5ccfa8a47d71b0151c4642c2a14269df Jul 20 23:30:00.559982 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:00.559958 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6"] Jul 20 23:30:00.635935 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:30:00.635900 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dff0a41_8053_4871_9c8c_9be9ec7ed0da.slice/crio-e5597d0c7d700663818b452b1ff93e0486e573e3a6d7ad638979835688e1bf3b WatchSource:0}: Error finding container e5597d0c7d700663818b452b1ff93e0486e573e3a6d7ad638979835688e1bf3b: Status 404 returned error can't find the container with id e5597d0c7d700663818b452b1ff93e0486e573e3a6d7ad638979835688e1bf3b Jul 20 23:30:01.271014 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.270913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" event={"ID":"2dff0a41-8053-4871-9c8c-9be9ec7ed0da","Type":"ContainerStarted","Data":"3a91437939e97134a2d30beeb0017c4aa82e962539b03ca4ce68f6610bf00293"} Jul 20 23:30:01.271014 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.270972 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" event={"ID":"2dff0a41-8053-4871-9c8c-9be9ec7ed0da","Type":"ContainerStarted","Data":"e5597d0c7d700663818b452b1ff93e0486e573e3a6d7ad638979835688e1bf3b"} Jul 20 23:30:01.272405 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.272382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" event={"ID":"0a29b6a3-9213-4fcf-b721-a10f10822d94","Type":"ContainerStarted","Data":"fee1791dbe025a6a17c8389382049ee5564c3b9f42afdf9d55bd14fd26d046d2"} Jul 20 23:30:01.272551 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.272426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" event={"ID":"0a29b6a3-9213-4fcf-b721-a10f10822d94","Type":"ContainerStarted","Data":"8498a243f18b6b10d9d2219cbc86f4db5ccfa8a47d71b0151c4642c2a14269df"} Jul 20 23:30:01.273870 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.273848 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" event={"ID":"22909265-657b-4701-a98c-cf6c6e52e605","Type":"ContainerStarted","Data":"b4977fc9ee53d757949b0a5dd1f34db491c6f0ba7a619b919d73a5688821e101"} Jul 20 23:30:01.273969 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.273891 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" event={"ID":"22909265-657b-4701-a98c-cf6c6e52e605","Type":"ContainerStarted","Data":"201c26d068f8b6b2ce8fa4a83d773710f2325ce08a96d3ea87ec2cae4a15a67c"} Jul 20 23:30:01.287293 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.287250 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" podStartSLOduration=1.287236439 podStartE2EDuration="1.287236439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:30:01.286745081 +0000 UTC m=+2190.783199497" watchObservedRunningTime="2026-07-20 23:30:01.287236439 +0000 UTC m=+2190.783690860" Jul 20 23:30:01.302477 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.302435 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" podStartSLOduration=1.302421393 podStartE2EDuration="1.302421393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:30:01.300140141 +0000 UTC m=+2190.796594576" watchObservedRunningTime="2026-07-20 23:30:01.302421393 +0000 UTC m=+2190.798875814" Jul 20 23:30:01.315052 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:01.315013 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" podStartSLOduration=1.315001236 podStartE2EDuration="1.315001236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:30:01.313906018 +0000 UTC m=+2190.810360441" watchObservedRunningTime="2026-07-20 23:30:01.315001236 +0000 UTC m=+2190.811455660" Jul 20 23:30:02.281372 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:02.281335 2574 generic.go:356] "Generic (PLEG): container finished" podID="0a29b6a3-9213-4fcf-b721-a10f10822d94" containerID="fee1791dbe025a6a17c8389382049ee5564c3b9f42afdf9d55bd14fd26d046d2" exitCode=0 Jul 20 23:30:02.281863 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:02.281418 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" event={"ID":"0a29b6a3-9213-4fcf-b721-a10f10822d94","Type":"ContainerDied","Data":"fee1791dbe025a6a17c8389382049ee5564c3b9f42afdf9d55bd14fd26d046d2"} Jul 20 23:30:02.283000 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:02.282978 2574 generic.go:356] "Generic (PLEG): container finished" podID="22909265-657b-4701-a98c-cf6c6e52e605" containerID="b4977fc9ee53d757949b0a5dd1f34db491c6f0ba7a619b919d73a5688821e101" exitCode=0 Jul 20 23:30:02.283109 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:02.283059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" event={"ID":"22909265-657b-4701-a98c-cf6c6e52e605","Type":"ContainerDied","Data":"b4977fc9ee53d757949b0a5dd1f34db491c6f0ba7a619b919d73a5688821e101"} Jul 20 23:30:02.284610 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:02.284583 2574 generic.go:356] "Generic (PLEG): container finished" podID="2dff0a41-8053-4871-9c8c-9be9ec7ed0da" containerID="3a91437939e97134a2d30beeb0017c4aa82e962539b03ca4ce68f6610bf00293" exitCode=0 Jul 20 23:30:02.284724 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:02.284630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" event={"ID":"2dff0a41-8053-4871-9c8c-9be9ec7ed0da","Type":"ContainerDied","Data":"3a91437939e97134a2d30beeb0017c4aa82e962539b03ca4ce68f6610bf00293"} Jul 20 23:30:03.431552 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:03.431524 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" Jul 20 23:30:03.476942 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:03.476916 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" Jul 20 23:30:03.479901 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:03.479883 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" Jul 20 23:30:04.295240 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.295212 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" Jul 20 23:30:04.295442 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.295249 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6" event={"ID":"2dff0a41-8053-4871-9c8c-9be9ec7ed0da","Type":"ContainerDied","Data":"e5597d0c7d700663818b452b1ff93e0486e573e3a6d7ad638979835688e1bf3b"} Jul 20 23:30:04.295442 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.295306 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5597d0c7d700663818b452b1ff93e0486e573e3a6d7ad638979835688e1bf3b" Jul 20 23:30:04.296663 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.296646 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" Jul 20 23:30:04.296782 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.296662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743170-hlngt" event={"ID":"0a29b6a3-9213-4fcf-b721-a10f10822d94","Type":"ContainerDied","Data":"8498a243f18b6b10d9d2219cbc86f4db5ccfa8a47d71b0151c4642c2a14269df"} Jul 20 23:30:04.296782 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.296701 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8498a243f18b6b10d9d2219cbc86f4db5ccfa8a47d71b0151c4642c2a14269df" Jul 20 23:30:04.298114 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.298097 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" Jul 20 23:30:04.298209 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.298154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv" event={"ID":"22909265-657b-4701-a98c-cf6c6e52e605","Type":"ContainerDied","Data":"201c26d068f8b6b2ce8fa4a83d773710f2325ce08a96d3ea87ec2cae4a15a67c"} Jul 20 23:30:04.298209 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:04.298198 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="201c26d068f8b6b2ce8fa4a83d773710f2325ce08a96d3ea87ec2cae4a15a67c" Jul 20 23:30:07.010691 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.010661 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2"] Jul 20 23:30:07.134392 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134355 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld"] Jul 20 23:30:07.134796 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134776 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22909265-657b-4701-a98c-cf6c6e52e605" containerName="cleanup" Jul 20 23:30:07.134796 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134797 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22909265-657b-4701-a98c-cf6c6e52e605" containerName="cleanup" Jul 20 23:30:07.134951 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134814 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dff0a41-8053-4871-9c8c-9be9ec7ed0da" containerName="cleanup" Jul 20 23:30:07.134951 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134824 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dff0a41-8053-4871-9c8c-9be9ec7ed0da" containerName="cleanup" Jul 20 23:30:07.134951 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134845 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a29b6a3-9213-4fcf-b721-a10f10822d94" containerName="cleanup" Jul 20 23:30:07.134951 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134853 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a29b6a3-9213-4fcf-b721-a10f10822d94" containerName="cleanup" Jul 20 23:30:07.135147 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134969 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="22909265-657b-4701-a98c-cf6c6e52e605" containerName="cleanup" Jul 20 23:30:07.135147 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134982 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dff0a41-8053-4871-9c8c-9be9ec7ed0da" containerName="cleanup" Jul 20 23:30:07.135147 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.134997 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a29b6a3-9213-4fcf-b721-a10f10822d94" containerName="cleanup" Jul 20 23:30:07.138100 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.138077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" Jul 20 23:30:07.140785 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.140763 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:30:07.140965 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.140777 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:30:07.144437 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.144413 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld"] Jul 20 23:30:07.304557 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.304463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78ada4-8aa3-4837-b9a9-cee624315e32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld\" (UID: \"8c78ada4-8aa3-4837-b9a9-cee624315e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" Jul 20 23:30:07.405571 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.405535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78ada4-8aa3-4837-b9a9-cee624315e32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld\" (UID: \"8c78ada4-8aa3-4837-b9a9-cee624315e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" Jul 20 23:30:07.406146 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.406125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78ada4-8aa3-4837-b9a9-cee624315e32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld\" (UID: \"8c78ada4-8aa3-4837-b9a9-cee624315e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" Jul 20 23:30:07.449756 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.449727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" Jul 20 23:30:07.583708 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:07.583674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld"] Jul 20 23:30:07.585279 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:30:07.585252 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c78ada4_8aa3_4837_b9a9_cee624315e32.slice/crio-59067dc8e60cf4be1e2a4622a7d095e06bc7af13b3db441cb98d768ef529fde7 WatchSource:0}: Error finding container 59067dc8e60cf4be1e2a4622a7d095e06bc7af13b3db441cb98d768ef529fde7: Status 404 returned error can't find the container with id 59067dc8e60cf4be1e2a4622a7d095e06bc7af13b3db441cb98d768ef529fde7 Jul 20 23:30:08.325106 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:08.325015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" event={"ID":"8c78ada4-8aa3-4837-b9a9-cee624315e32","Type":"ContainerStarted","Data":"58b32b1c02e0c35c65a8997fffdf50025df426acdfaacd67921b077534cf4660"} Jul 20 23:30:08.325106 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:08.325073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" event={"ID":"8c78ada4-8aa3-4837-b9a9-cee624315e32","Type":"ContainerStarted","Data":"59067dc8e60cf4be1e2a4622a7d095e06bc7af13b3db441cb98d768ef529fde7"} Jul 20 23:30:08.340174 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:08.340127 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" podStartSLOduration=1.3401088460000001 podStartE2EDuration="1.340108846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:30:08.339309657 +0000 UTC m=+2197.835764082" watchObservedRunningTime="2026-07-20 23:30:08.340108846 +0000 UTC m=+2197.836563270" Jul 20 23:30:09.331998 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:09.331963 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c78ada4-8aa3-4837-b9a9-cee624315e32" containerID="58b32b1c02e0c35c65a8997fffdf50025df426acdfaacd67921b077534cf4660" exitCode=0 Jul 20 23:30:09.332434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:09.332055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" event={"ID":"8c78ada4-8aa3-4837-b9a9-cee624315e32","Type":"ContainerDied","Data":"58b32b1c02e0c35c65a8997fffdf50025df426acdfaacd67921b077534cf4660"} Jul 20 23:30:10.469773 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:10.469744 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" Jul 20 23:30:10.634649 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:10.634612 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78ada4-8aa3-4837-b9a9-cee624315e32-maas-api-service-ca\") pod \"8c78ada4-8aa3-4837-b9a9-cee624315e32\" (UID: \"8c78ada4-8aa3-4837-b9a9-cee624315e32\") " Jul 20 23:30:10.634988 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:10.634963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c78ada4-8aa3-4837-b9a9-cee624315e32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c78ada4-8aa3-4837-b9a9-cee624315e32" (UID: "8c78ada4-8aa3-4837-b9a9-cee624315e32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:30:10.735733 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:10.735698 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c78ada4-8aa3-4837-b9a9-cee624315e32-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:11.073202 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:11.073162 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:30:11.345690 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:11.345663 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" Jul 20 23:30:11.345861 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:11.345664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld" event={"ID":"8c78ada4-8aa3-4837-b9a9-cee624315e32","Type":"ContainerDied","Data":"59067dc8e60cf4be1e2a4622a7d095e06bc7af13b3db441cb98d768ef529fde7"} Jul 20 23:30:11.345861 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:11.345767 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59067dc8e60cf4be1e2a4622a7d095e06bc7af13b3db441cb98d768ef529fde7" Jul 20 23:30:17.559926 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:17.559887 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6"] Jul 20 23:30:17.562027 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:17.562002 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6a0c93-29743170-5hkv6"] Jul 20 23:30:19.008802 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:19.008769 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dff0a41-8053-4871-9c8c-9be9ec7ed0da" path="/var/lib/kubelet/pods/2dff0a41-8053-4871-9c8c-9be9ec7ed0da/volumes" Jul 20 23:30:22.030716 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.030664 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" podUID="1bc78569-94cb-49c0-94fc-eb1d73356aac" containerName="main" containerID="cri-o://dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03" gracePeriod=45 Jul 20 23:30:22.220408 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.220367 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld"] Jul 20 23:30:22.223290 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.223260 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6a0c93-wztld"] Jul 20 23:30:22.291119 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.291058 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:30:22.334144 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334114 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-model-cache\") pod \"1bc78569-94cb-49c0-94fc-eb1d73356aac\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " Jul 20 23:30:22.334144 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334145 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-home\") pod \"1bc78569-94cb-49c0-94fc-eb1d73356aac\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " Jul 20 23:30:22.334364 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334178 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md4hw\" (UniqueName: \"kubernetes.io/projected/1bc78569-94cb-49c0-94fc-eb1d73356aac-kube-api-access-md4hw\") pod \"1bc78569-94cb-49c0-94fc-eb1d73356aac\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " Jul 20 23:30:22.334364 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-dshm\") pod \"1bc78569-94cb-49c0-94fc-eb1d73356aac\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " Jul 20 23:30:22.334364 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334275 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-tmp-dir\") pod \"1bc78569-94cb-49c0-94fc-eb1d73356aac\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " Jul 20 23:30:22.334364 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334296 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc78569-94cb-49c0-94fc-eb1d73356aac-tls-certs\") pod \"1bc78569-94cb-49c0-94fc-eb1d73356aac\" (UID: \"1bc78569-94cb-49c0-94fc-eb1d73356aac\") " Jul 20 23:30:22.334598 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334432 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-home" (OuterVolumeSpecName: "home") pod "1bc78569-94cb-49c0-94fc-eb1d73356aac" (UID: "1bc78569-94cb-49c0-94fc-eb1d73356aac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:22.334598 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334436 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bc78569-94cb-49c0-94fc-eb1d73356aac" (UID: "1bc78569-94cb-49c0-94fc-eb1d73356aac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:22.334697 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334606 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bc78569-94cb-49c0-94fc-eb1d73356aac" (UID: "1bc78569-94cb-49c0-94fc-eb1d73356aac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:22.334697 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334613 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-model-cache\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:22.334697 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.334655 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-home\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:22.336665 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.336640 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-dshm" (OuterVolumeSpecName: "dshm") pod "1bc78569-94cb-49c0-94fc-eb1d73356aac" (UID: "1bc78569-94cb-49c0-94fc-eb1d73356aac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:30:22.336776 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.336706 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc78569-94cb-49c0-94fc-eb1d73356aac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bc78569-94cb-49c0-94fc-eb1d73356aac" (UID: "1bc78569-94cb-49c0-94fc-eb1d73356aac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:30:22.336829 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.336778 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc78569-94cb-49c0-94fc-eb1d73356aac-kube-api-access-md4hw" (OuterVolumeSpecName: "kube-api-access-md4hw") pod "1bc78569-94cb-49c0-94fc-eb1d73356aac" (UID: "1bc78569-94cb-49c0-94fc-eb1d73356aac"). InnerVolumeSpecName "kube-api-access-md4hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:30:22.409870 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.409836 2574 generic.go:356] "Generic (PLEG): container finished" podID="1bc78569-94cb-49c0-94fc-eb1d73356aac" containerID="dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03" exitCode=0 Jul 20 23:30:22.410018 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.409900 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" Jul 20 23:30:22.410018 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.409925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" event={"ID":"1bc78569-94cb-49c0-94fc-eb1d73356aac","Type":"ContainerDied","Data":"dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03"} Jul 20 23:30:22.410018 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.409970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2" event={"ID":"1bc78569-94cb-49c0-94fc-eb1d73356aac","Type":"ContainerDied","Data":"366c1bac460e3740fa93736db5f2f9381cf8cfe9f37ebd8fe9077bdf72916b33"} Jul 20 23:30:22.410018 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.409989 2574 scope.go:117] "RemoveContainer" containerID="dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03" Jul 20 23:30:22.419347 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.419330 2574 scope.go:117] "RemoveContainer" containerID="dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03" Jul 20 23:30:22.419662 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:30:22.419642 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03\": container with ID starting with dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03 not found: ID does not exist" containerID="dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03" Jul 20 23:30:22.419744 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.419673 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03"} err="failed to get container status \"dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03\": rpc error: code = NotFound desc = could not find container \"dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03\": container with ID starting with dbb5d629a573fa4850f6dcf75c65da7136693907012a8591493d45b62bb25a03 not found: ID does not exist" Jul 20 23:30:22.432301 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.432277 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2"] Jul 20 23:30:22.435807 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.435787 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md4hw\" (UniqueName: \"kubernetes.io/projected/1bc78569-94cb-49c0-94fc-eb1d73356aac-kube-api-access-md4hw\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:22.435807 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.435807 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-dshm\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:22.435934 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.435818 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc78569-94cb-49c0-94fc-eb1d73356aac-tmp-dir\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:22.435934 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.435827 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc78569-94cb-49c0-94fc-eb1d73356aac-tls-certs\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:22.435934 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:22.435916 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-324c54/test-model-324c54-kserve-68b9bdff5f-qf2h2"] Jul 20 23:30:23.009609 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:23.009570 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bc78569-94cb-49c0-94fc-eb1d73356aac" path="/var/lib/kubelet/pods/1bc78569-94cb-49c0-94fc-eb1d73356aac/volumes" Jul 20 23:30:23.009987 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:23.009967 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c78ada4-8aa3-4837-b9a9-cee624315e32" path="/var/lib/kubelet/pods/8c78ada4-8aa3-4837-b9a9-cee624315e32/volumes" Jul 20 23:30:27.079113 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:27.079077 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:30:29.067357 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.067318 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz"] Jul 20 23:30:29.067867 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.067836 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c78ada4-8aa3-4837-b9a9-cee624315e32" containerName="revoke-keys" Jul 20 23:30:29.067867 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.067857 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c78ada4-8aa3-4837-b9a9-cee624315e32" containerName="revoke-keys" Jul 20 23:30:29.067992 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.067902 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc78569-94cb-49c0-94fc-eb1d73356aac" containerName="main" Jul 20 23:30:29.067992 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.067912 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc78569-94cb-49c0-94fc-eb1d73356aac" containerName="main" Jul 20 23:30:29.068089 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.068011 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c78ada4-8aa3-4837-b9a9-cee624315e32" containerName="revoke-keys" Jul 20 23:30:29.068089 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.068027 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc78569-94cb-49c0-94fc-eb1d73356aac" containerName="main" Jul 20 23:30:29.072426 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.072404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" Jul 20 23:30:29.075046 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.075024 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:30:29.075618 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.075589 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:30:29.077925 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.077897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz"] Jul 20 23:30:29.195729 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.195690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e897b4-89b9-4a82-b097-1a0c02aa82bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz\" (UID: \"02e897b4-89b9-4a82-b097-1a0c02aa82bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" Jul 20 23:30:29.296280 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.296241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e897b4-89b9-4a82-b097-1a0c02aa82bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz\" (UID: \"02e897b4-89b9-4a82-b097-1a0c02aa82bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" Jul 20 23:30:29.296897 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.296877 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e897b4-89b9-4a82-b097-1a0c02aa82bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz\" (UID: \"02e897b4-89b9-4a82-b097-1a0c02aa82bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" Jul 20 23:30:29.383801 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.383766 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" Jul 20 23:30:29.542744 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:29.542709 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz"] Jul 20 23:30:29.544437 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:30:29.544409 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e897b4_89b9_4a82_b097_1a0c02aa82bf.slice/crio-df6268e6c21a9c74c76bc2cff6d57692f0de287333c89e135a2464b3854dabd3 WatchSource:0}: Error finding container df6268e6c21a9c74c76bc2cff6d57692f0de287333c89e135a2464b3854dabd3: Status 404 returned error can't find the container with id df6268e6c21a9c74c76bc2cff6d57692f0de287333c89e135a2464b3854dabd3 Jul 20 23:30:30.471485 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:30.471450 2574 generic.go:356] "Generic (PLEG): container finished" podID="02e897b4-89b9-4a82-b097-1a0c02aa82bf" containerID="99a76b64990c440e2e5ea420c4c2c6d3092703f959eea449accf3181a9a22143" exitCode=0 Jul 20 23:30:30.471876 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:30.471539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" event={"ID":"02e897b4-89b9-4a82-b097-1a0c02aa82bf","Type":"ContainerDied","Data":"99a76b64990c440e2e5ea420c4c2c6d3092703f959eea449accf3181a9a22143"} Jul 20 23:30:30.471876 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:30.471586 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" event={"ID":"02e897b4-89b9-4a82-b097-1a0c02aa82bf","Type":"ContainerStarted","Data":"df6268e6c21a9c74c76bc2cff6d57692f0de287333c89e135a2464b3854dabd3"} Jul 20 23:30:31.614314 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:31.614285 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" Jul 20 23:30:31.720405 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:31.720375 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e897b4-89b9-4a82-b097-1a0c02aa82bf-maas-api-service-ca\") pod \"02e897b4-89b9-4a82-b097-1a0c02aa82bf\" (UID: \"02e897b4-89b9-4a82-b097-1a0c02aa82bf\") " Jul 20 23:30:31.720758 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:31.720735 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e897b4-89b9-4a82-b097-1a0c02aa82bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02e897b4-89b9-4a82-b097-1a0c02aa82bf" (UID: "02e897b4-89b9-4a82-b097-1a0c02aa82bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:30:31.821169 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:31.821075 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e897b4-89b9-4a82-b097-1a0c02aa82bf-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:30:32.484335 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:32.484306 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" Jul 20 23:30:32.484335 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:32.484325 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz" event={"ID":"02e897b4-89b9-4a82-b097-1a0c02aa82bf","Type":"ContainerDied","Data":"df6268e6c21a9c74c76bc2cff6d57692f0de287333c89e135a2464b3854dabd3"} Jul 20 23:30:32.484586 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:32.484368 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6268e6c21a9c74c76bc2cff6d57692f0de287333c89e135a2464b3854dabd3" Jul 20 23:30:39.456164 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:39.456131 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv"] Jul 20 23:30:39.458942 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:39.458921 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-324c54-29743170-4stwv"] Jul 20 23:30:41.009196 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:41.009162 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22909265-657b-4701-a98c-cf6c6e52e605" path="/var/lib/kubelet/pods/22909265-657b-4701-a98c-cf6c6e52e605/volumes" Jul 20 23:30:42.278397 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:42.278366 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:30:44.138682 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:44.138648 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz"] Jul 20 23:30:44.141847 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:44.141821 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-324c54-qp8vz"] Jul 20 23:30:45.008901 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:30:45.008872 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e897b4-89b9-4a82-b097-1a0c02aa82bf" path="/var/lib/kubelet/pods/02e897b4-89b9-4a82-b097-1a0c02aa82bf/volumes" Jul 20 23:31:01.579346 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:01.579263 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:31:14.581840 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.581803 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5"] Jul 20 23:31:14.582335 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.582138 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e897b4-89b9-4a82-b097-1a0c02aa82bf" containerName="revoke-keys" Jul 20 23:31:14.582335 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.582149 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e897b4-89b9-4a82-b097-1a0c02aa82bf" containerName="revoke-keys" Jul 20 23:31:14.582335 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.582224 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e897b4-89b9-4a82-b097-1a0c02aa82bf" containerName="revoke-keys" Jul 20 23:31:14.585099 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.585077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" Jul 20 23:31:14.587258 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.587239 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:31:14.587383 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.587365 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:31:14.593327 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.593301 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5"] Jul 20 23:31:14.686160 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.686132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fa5742-d13f-45cd-99a6-cc32505858b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5\" (UID: \"39fa5742-d13f-45cd-99a6-cc32505858b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" Jul 20 23:31:14.787568 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.787534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fa5742-d13f-45cd-99a6-cc32505858b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5\" (UID: \"39fa5742-d13f-45cd-99a6-cc32505858b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" Jul 20 23:31:14.788163 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.788145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fa5742-d13f-45cd-99a6-cc32505858b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5\" (UID: \"39fa5742-d13f-45cd-99a6-cc32505858b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" Jul 20 23:31:14.897254 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:14.897226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" Jul 20 23:31:15.031829 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:15.031796 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5"] Jul 20 23:31:15.033667 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:31:15.033635 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fa5742_d13f_45cd_99a6_cc32505858b8.slice/crio-2974ab7d07d93e678f0d63cfb1ce1925853f42396f898bf1e6a7cc733e18a127 WatchSource:0}: Error finding container 2974ab7d07d93e678f0d63cfb1ce1925853f42396f898bf1e6a7cc733e18a127: Status 404 returned error can't find the container with id 2974ab7d07d93e678f0d63cfb1ce1925853f42396f898bf1e6a7cc733e18a127 Jul 20 23:31:15.722989 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:15.722953 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" event={"ID":"39fa5742-d13f-45cd-99a6-cc32505858b8","Type":"ContainerStarted","Data":"fcb510930726056f17110af6182f8188dc88bfb055a155cbe4ec9ed63f411aea"} Jul 20 23:31:15.723413 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:15.723014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" event={"ID":"39fa5742-d13f-45cd-99a6-cc32505858b8","Type":"ContainerStarted","Data":"2974ab7d07d93e678f0d63cfb1ce1925853f42396f898bf1e6a7cc733e18a127"} Jul 20 23:31:15.742587 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:15.742537 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" podStartSLOduration=1.742522111 podStartE2EDuration="1.742522111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:31:15.739324527 +0000 UTC m=+2265.235778965" watchObservedRunningTime="2026-07-20 23:31:15.742522111 +0000 UTC m=+2265.238976532" Jul 20 23:31:16.729681 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:16.729644 2574 generic.go:356] "Generic (PLEG): container finished" podID="39fa5742-d13f-45cd-99a6-cc32505858b8" containerID="fcb510930726056f17110af6182f8188dc88bfb055a155cbe4ec9ed63f411aea" exitCode=0 Jul 20 23:31:16.730090 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:16.729696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" event={"ID":"39fa5742-d13f-45cd-99a6-cc32505858b8","Type":"ContainerDied","Data":"fcb510930726056f17110af6182f8188dc88bfb055a155cbe4ec9ed63f411aea"} Jul 20 23:31:17.863201 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:17.863178 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" Jul 20 23:31:17.912608 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:17.912579 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fa5742-d13f-45cd-99a6-cc32505858b8-maas-api-service-ca\") pod \"39fa5742-d13f-45cd-99a6-cc32505858b8\" (UID: \"39fa5742-d13f-45cd-99a6-cc32505858b8\") " Jul 20 23:31:17.912953 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:17.912932 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fa5742-d13f-45cd-99a6-cc32505858b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39fa5742-d13f-45cd-99a6-cc32505858b8" (UID: "39fa5742-d13f-45cd-99a6-cc32505858b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:31:18.014003 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:18.013927 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39fa5742-d13f-45cd-99a6-cc32505858b8-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:31:18.747514 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:18.747462 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" Jul 20 23:31:18.747687 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:18.747513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5" event={"ID":"39fa5742-d13f-45cd-99a6-cc32505858b8","Type":"ContainerDied","Data":"2974ab7d07d93e678f0d63cfb1ce1925853f42396f898bf1e6a7cc733e18a127"} Jul 20 23:31:18.747687 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:18.747586 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2974ab7d07d93e678f0d63cfb1ce1925853f42396f898bf1e6a7cc733e18a127" Jul 20 23:31:19.176443 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:19.176408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:31:31.212410 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:31.212376 2574 scope.go:117] "RemoveContainer" containerID="ee4b3f5fb28ce2de431ea5a1966783f760774c15065afe9c79cc0e185078497d" Jul 20 23:31:34.666046 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:34.665998 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5"] Jul 20 23:31:34.668309 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:34.668288 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4462e-6kvx5"] Jul 20 23:31:35.008927 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:35.008854 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39fa5742-d13f-45cd-99a6-cc32505858b8" path="/var/lib/kubelet/pods/39fa5742-d13f-45cd-99a6-cc32505858b8/volumes" Jul 20 23:31:36.166147 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:36.166107 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:31:41.320950 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.320917 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl"] Jul 20 23:31:41.321451 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.321430 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39fa5742-d13f-45cd-99a6-cc32505858b8" containerName="revoke-keys" Jul 20 23:31:41.321575 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.321455 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="39fa5742-d13f-45cd-99a6-cc32505858b8" containerName="revoke-keys" Jul 20 23:31:41.321575 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.321564 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="39fa5742-d13f-45cd-99a6-cc32505858b8" containerName="revoke-keys" Jul 20 23:31:41.324605 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.324585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" Jul 20 23:31:41.326732 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.326697 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:31:41.326833 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.326775 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:31:41.331730 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.331705 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl"] Jul 20 23:31:41.418404 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.418375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728d3312-8a14-4d53-806e-9e6d5c820636-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl\" (UID: \"728d3312-8a14-4d53-806e-9e6d5c820636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" Jul 20 23:31:41.519555 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.519525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728d3312-8a14-4d53-806e-9e6d5c820636-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl\" (UID: \"728d3312-8a14-4d53-806e-9e6d5c820636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" Jul 20 23:31:41.520082 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.520062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728d3312-8a14-4d53-806e-9e6d5c820636-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl\" (UID: \"728d3312-8a14-4d53-806e-9e6d5c820636\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" Jul 20 23:31:41.635508 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.635464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" Jul 20 23:31:41.765626 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.765597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl"] Jul 20 23:31:41.766872 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:31:41.766847 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728d3312_8a14_4d53_806e_9e6d5c820636.slice/crio-f90aba7a2a07ff22a0cc9c221b37d1fd3f75e7e2e6af349e0f8b1ab637f73b11 WatchSource:0}: Error finding container f90aba7a2a07ff22a0cc9c221b37d1fd3f75e7e2e6af349e0f8b1ab637f73b11: Status 404 returned error can't find the container with id f90aba7a2a07ff22a0cc9c221b37d1fd3f75e7e2e6af349e0f8b1ab637f73b11 Jul 20 23:31:41.870248 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:41.870213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" event={"ID":"728d3312-8a14-4d53-806e-9e6d5c820636","Type":"ContainerStarted","Data":"f90aba7a2a07ff22a0cc9c221b37d1fd3f75e7e2e6af349e0f8b1ab637f73b11"} Jul 20 23:31:42.882763 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:42.882726 2574 generic.go:356] "Generic (PLEG): container finished" podID="728d3312-8a14-4d53-806e-9e6d5c820636" containerID="c0b92eacb8e8f896adaff233b0bbbbefad75aab2a801a3837058424e1df81e3e" exitCode=0 Jul 20 23:31:42.883157 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:42.882767 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" event={"ID":"728d3312-8a14-4d53-806e-9e6d5c820636","Type":"ContainerDied","Data":"c0b92eacb8e8f896adaff233b0bbbbefad75aab2a801a3837058424e1df81e3e"} Jul 20 23:31:44.022762 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:44.022739 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" Jul 20 23:31:44.142021 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:44.141949 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728d3312-8a14-4d53-806e-9e6d5c820636-maas-api-service-ca\") pod \"728d3312-8a14-4d53-806e-9e6d5c820636\" (UID: \"728d3312-8a14-4d53-806e-9e6d5c820636\") " Jul 20 23:31:44.142317 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:44.142296 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728d3312-8a14-4d53-806e-9e6d5c820636-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "728d3312-8a14-4d53-806e-9e6d5c820636" (UID: "728d3312-8a14-4d53-806e-9e6d5c820636"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:31:44.243357 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:44.243324 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728d3312-8a14-4d53-806e-9e6d5c820636-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:31:44.894989 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:44.894961 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" Jul 20 23:31:44.895142 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:44.894968 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl" event={"ID":"728d3312-8a14-4d53-806e-9e6d5c820636","Type":"ContainerDied","Data":"f90aba7a2a07ff22a0cc9c221b37d1fd3f75e7e2e6af349e0f8b1ab637f73b11"} Jul 20 23:31:44.895142 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:31:44.895072 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f90aba7a2a07ff22a0cc9c221b37d1fd3f75e7e2e6af349e0f8b1ab637f73b11" Jul 20 23:32:01.394459 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:01.394414 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl"] Jul 20 23:32:01.396892 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:01.396867 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4c2828-xtkcl"] Jul 20 23:32:01.874734 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:01.874694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:32:03.009347 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:03.009312 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="728d3312-8a14-4d53-806e-9e6d5c820636" path="/var/lib/kubelet/pods/728d3312-8a14-4d53-806e-9e6d5c820636/volumes" Jul 20 23:32:17.175094 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:17.175058 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:32:25.472601 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:25.472518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:32:31.235918 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:31.235884 2574 scope.go:117] "RemoveContainer" containerID="6a0c4717b995f4f3f579fa65f67ff725d9d9d30c6dc3bebf59944971eca1a484" Jul 20 23:32:48.781605 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:32:48.781558 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:33:13.326389 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.326344 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf"] Jul 20 23:33:13.326895 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.326724 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="728d3312-8a14-4d53-806e-9e6d5c820636" containerName="revoke-keys" Jul 20 23:33:13.326895 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.326736 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="728d3312-8a14-4d53-806e-9e6d5c820636" containerName="revoke-keys" Jul 20 23:33:13.326895 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.326808 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="728d3312-8a14-4d53-806e-9e6d5c820636" containerName="revoke-keys" Jul 20 23:33:13.329729 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.329713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.332866 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.332843 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"auth-test-model-5b9afe-kserve-self-signed-certs\"" Jul 20 23:33:13.332995 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.332845 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"default-dockercfg-v9wvl\"" Jul 20 23:33:13.332995 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.332881 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"openshift-service-ca.crt\"" Jul 20 23:33:13.332995 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.332845 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"kube-root-ca.crt\"" Jul 20 23:33:13.337968 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.337558 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf"] Jul 20 23:33:13.359172 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.359145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30271354-a3fa-4601-a921-fc727a201255-tls-certs\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.359326 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.359186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-model-cache\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.359326 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.359208 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-tmp-dir\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.359326 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.359251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-home\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.359326 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.359273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbkq\" (UniqueName: \"kubernetes.io/projected/30271354-a3fa-4601-a921-fc727a201255-kube-api-access-mbbkq\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.359326 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.359319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-dshm\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460333 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-dshm\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460541 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30271354-a3fa-4601-a921-fc727a201255-tls-certs\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460541 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-model-cache\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460541 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-tmp-dir\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460541 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-home\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460541 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbkq\" (UniqueName: \"kubernetes.io/projected/30271354-a3fa-4601-a921-fc727a201255-kube-api-access-mbbkq\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460828 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-model-cache\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460892 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-tmp-dir\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.460947 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.460891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-home\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.462639 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.462621 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-dshm\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.462933 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.462916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30271354-a3fa-4601-a921-fc727a201255-tls-certs\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.468529 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.468505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbkq\" (UniqueName: \"kubernetes.io/projected/30271354-a3fa-4601-a921-fc727a201255-kube-api-access-mbbkq\") pod \"auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.640848 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.640805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:13.781058 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:13.781028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf"] Jul 20 23:33:13.783048 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:33:13.783022 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30271354_a3fa_4601_a921_fc727a201255.slice/crio-3fbc56197035427fd57133f4e5c7cdc6ef68df2304a25765ae48d93019e6160a WatchSource:0}: Error finding container 3fbc56197035427fd57133f4e5c7cdc6ef68df2304a25765ae48d93019e6160a: Status 404 returned error can't find the container with id 3fbc56197035427fd57133f4e5c7cdc6ef68df2304a25765ae48d93019e6160a Jul 20 23:33:14.378026 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:14.377991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" event={"ID":"30271354-a3fa-4601-a921-fc727a201255","Type":"ContainerStarted","Data":"b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84"} Jul 20 23:33:14.378446 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:14.378042 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" event={"ID":"30271354-a3fa-4601-a921-fc727a201255","Type":"ContainerStarted","Data":"3fbc56197035427fd57133f4e5c7cdc6ef68df2304a25765ae48d93019e6160a"} Jul 20 23:33:14.393605 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:14.393553 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" podStartSLOduration=1.393535066 podStartE2EDuration="1.393535066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:33:14.392441801 +0000 UTC m=+2383.888896250" watchObservedRunningTime="2026-07-20 23:33:14.393535066 +0000 UTC m=+2383.889989490" Jul 20 23:33:17.490401 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:17.490360 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:33:23.641801 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:23.641754 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:23.642293 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:23.641821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:23.654548 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:23.654517 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:24.445247 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:24.445212 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:33:31.111586 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:31.111560 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:33:31.116900 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:31.116868 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:33:31.122131 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:31.122112 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:33:31.127297 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:33:31.127274 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:34:01.779958 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:34:01.779877 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:34:20.575675 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:34:20.575638 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:34:36.574974 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:34:36.574934 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:34:55.708894 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:34:55.708857 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:35:14.869283 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:35:14.869241 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:35:34.074271 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:35:34.074184 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:35:51.568865 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:35:51.568829 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:36:05.401714 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:05.401677 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf"] Jul 20 23:36:07.975336 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:07.975305 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:36:17.381301 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:17.381261 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:36:20.422254 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.422186 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" podUID="30271354-a3fa-4601-a921-fc727a201255" containerName="main" containerID="cri-o://b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84" gracePeriod=45 Jul 20 23:36:20.674840 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.674779 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:36:20.724598 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-tmp-dir\") pod \"30271354-a3fa-4601-a921-fc727a201255\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " Jul 20 23:36:20.724769 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724664 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-dshm\") pod \"30271354-a3fa-4601-a921-fc727a201255\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " Jul 20 23:36:20.724769 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724696 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-model-cache\") pod \"30271354-a3fa-4601-a921-fc727a201255\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " Jul 20 23:36:20.724769 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724712 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-home\") pod \"30271354-a3fa-4601-a921-fc727a201255\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " Jul 20 23:36:20.724769 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724731 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbbkq\" (UniqueName: \"kubernetes.io/projected/30271354-a3fa-4601-a921-fc727a201255-kube-api-access-mbbkq\") pod \"30271354-a3fa-4601-a921-fc727a201255\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " Jul 20 23:36:20.724769 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724749 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30271354-a3fa-4601-a921-fc727a201255-tls-certs\") pod \"30271354-a3fa-4601-a921-fc727a201255\" (UID: \"30271354-a3fa-4601-a921-fc727a201255\") " Jul 20 23:36:20.725031 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30271354-a3fa-4601-a921-fc727a201255" (UID: "30271354-a3fa-4601-a921-fc727a201255"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:20.725031 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724984 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-home" (OuterVolumeSpecName: "home") pod "30271354-a3fa-4601-a921-fc727a201255" (UID: "30271354-a3fa-4601-a921-fc727a201255"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:20.725031 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.724992 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-model-cache" (OuterVolumeSpecName: "model-cache") pod "30271354-a3fa-4601-a921-fc727a201255" (UID: "30271354-a3fa-4601-a921-fc727a201255"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:20.725031 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.725008 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-tmp-dir\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.726982 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.726956 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-dshm" (OuterVolumeSpecName: "dshm") pod "30271354-a3fa-4601-a921-fc727a201255" (UID: "30271354-a3fa-4601-a921-fc727a201255"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:36:20.727094 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.727010 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30271354-a3fa-4601-a921-fc727a201255-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30271354-a3fa-4601-a921-fc727a201255" (UID: "30271354-a3fa-4601-a921-fc727a201255"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:36:20.727094 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.727058 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30271354-a3fa-4601-a921-fc727a201255-kube-api-access-mbbkq" (OuterVolumeSpecName: "kube-api-access-mbbkq") pod "30271354-a3fa-4601-a921-fc727a201255" (UID: "30271354-a3fa-4601-a921-fc727a201255"). InnerVolumeSpecName "kube-api-access-mbbkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:36:20.826229 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.826197 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-dshm\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.826229 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.826223 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-model-cache\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.826229 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.826233 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30271354-a3fa-4601-a921-fc727a201255-home\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.826463 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.826242 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbkq\" (UniqueName: \"kubernetes.io/projected/30271354-a3fa-4601-a921-fc727a201255-kube-api-access-mbbkq\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:36:20.826463 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:20.826252 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30271354-a3fa-4601-a921-fc727a201255-tls-certs\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:36:21.406211 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.406178 2574 generic.go:356] "Generic (PLEG): container finished" podID="30271354-a3fa-4601-a921-fc727a201255" containerID="b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84" exitCode=0 Jul 20 23:36:21.406211 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.406215 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" Jul 20 23:36:21.406440 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.406223 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" event={"ID":"30271354-a3fa-4601-a921-fc727a201255","Type":"ContainerDied","Data":"b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84"} Jul 20 23:36:21.406440 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.406261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf" event={"ID":"30271354-a3fa-4601-a921-fc727a201255","Type":"ContainerDied","Data":"3fbc56197035427fd57133f4e5c7cdc6ef68df2304a25765ae48d93019e6160a"} Jul 20 23:36:21.406440 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.406281 2574 scope.go:117] "RemoveContainer" containerID="b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84" Jul 20 23:36:21.415261 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.415242 2574 scope.go:117] "RemoveContainer" containerID="b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84" Jul 20 23:36:21.415545 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:36:21.415524 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84\": container with ID starting with b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84 not found: ID does not exist" containerID="b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84" Jul 20 23:36:21.415631 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.415552 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84"} err="failed to get container status \"b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84\": rpc error: code = NotFound desc = could not find container \"b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84\": container with ID starting with b7e93595b3114fccb7c7657f9189c4b4061c163f03e827b18e8c30cb4f7b0f84 not found: ID does not exist" Jul 20 23:36:21.422049 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.422026 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf"] Jul 20 23:36:21.425950 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:21.425931 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/auth-test-model-5b9afe-kserve-b7dd5f88c-cm5jf"] Jul 20 23:36:23.009459 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:23.009426 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30271354-a3fa-4601-a921-fc727a201255" path="/var/lib/kubelet/pods/30271354-a3fa-4601-a921-fc727a201255/volumes" Jul 20 23:36:31.219013 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.218981 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g"] Jul 20 23:36:31.219446 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.219336 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30271354-a3fa-4601-a921-fc727a201255" containerName="main" Jul 20 23:36:31.219446 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.219348 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="30271354-a3fa-4601-a921-fc727a201255" containerName="main" Jul 20 23:36:31.219446 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.219420 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="30271354-a3fa-4601-a921-fc727a201255" containerName="main" Jul 20 23:36:31.223609 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.223591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.226089 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.226064 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"kube-root-ca.crt\"" Jul 20 23:36:31.226889 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.226867 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"default-dockercfg-v9wvl\"" Jul 20 23:36:31.226889 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.226879 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"openshift-service-ca.crt\"" Jul 20 23:36:31.227072 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.226909 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"sub-test-model-5b9afe-kserve-self-signed-certs\"" Jul 20 23:36:31.231764 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.231740 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g"] Jul 20 23:36:31.283883 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.283858 2574 scope.go:117] "RemoveContainer" containerID="99a76b64990c440e2e5ea420c4c2c6d3092703f959eea449accf3181a9a22143" Jul 20 23:36:31.292706 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.292689 2574 scope.go:117] "RemoveContainer" containerID="b4977fc9ee53d757949b0a5dd1f34db491c6f0ba7a619b919d73a5688821e101" Jul 20 23:36:31.301193 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.301164 2574 scope.go:117] "RemoveContainer" containerID="58b32b1c02e0c35c65a8997fffdf50025df426acdfaacd67921b077534cf4660" Jul 20 23:36:31.310811 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.310792 2574 scope.go:117] "RemoveContainer" containerID="3a91437939e97134a2d30beeb0017c4aa82e962539b03ca4ce68f6610bf00293" Jul 20 23:36:31.316279 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.316250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tls-certs\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.316375 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.316290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfg9\" (UniqueName: \"kubernetes.io/projected/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-kube-api-access-6qfg9\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.316375 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.316355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-model-cache\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.316469 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.316439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-dshm\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.316541 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.316509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tmp-dir\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.316585 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.316549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-home\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.417602 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.417570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tls-certs\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.417741 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.417611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfg9\" (UniqueName: \"kubernetes.io/projected/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-kube-api-access-6qfg9\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.417741 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.417645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-model-cache\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.417824 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.417756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-dshm\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.417824 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.417804 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tmp-dir\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.417893 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.417851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-home\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.418026 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.418007 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-model-cache\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.418160 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.418143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-home\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.418220 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.418155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tmp-dir\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.419869 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.419850 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-dshm\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.420136 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.420120 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tls-certs\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.427993 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.427973 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfg9\" (UniqueName: \"kubernetes.io/projected/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-kube-api-access-6qfg9\") pod \"sub-test-model-5b9afe-kserve-69c5496855-z2h2g\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.533841 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.533776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:31.662627 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:31.662596 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g"] Jul 20 23:36:31.666121 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:36:31.666096 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2cc7068_d367_48c6_a8b0_9c5edc639fa2.slice/crio-36fd5b9c2ea889cdf96b2d1c1d1dbcbcdab376cd9f8ccaf06b2bd6d2b962ebc8 WatchSource:0}: Error finding container 36fd5b9c2ea889cdf96b2d1c1d1dbcbcdab376cd9f8ccaf06b2bd6d2b962ebc8: Status 404 returned error can't find the container with id 36fd5b9c2ea889cdf96b2d1c1d1dbcbcdab376cd9f8ccaf06b2bd6d2b962ebc8 Jul 20 23:36:32.466848 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:32.466811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" event={"ID":"f2cc7068-d367-48c6-a8b0-9c5edc639fa2","Type":"ContainerStarted","Data":"4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4"} Jul 20 23:36:32.467248 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:32.466860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" event={"ID":"f2cc7068-d367-48c6-a8b0-9c5edc639fa2","Type":"ContainerStarted","Data":"36fd5b9c2ea889cdf96b2d1c1d1dbcbcdab376cd9f8ccaf06b2bd6d2b962ebc8"} Jul 20 23:36:32.510868 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:32.510819 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" podStartSLOduration=1.510804323 podStartE2EDuration="1.510804323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:36:32.507766054 +0000 UTC m=+2582.004220478" watchObservedRunningTime="2026-07-20 23:36:32.510804323 +0000 UTC m=+2582.007258757" Jul 20 23:36:34.974002 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:34.973965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:36:41.534069 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:41.534039 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:41.534470 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:41.534106 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:41.546276 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:41.546256 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:42.531848 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:42.531814 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:36:52.177439 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:36:52.177395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:37:09.217388 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:09.217317 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g"] Jul 20 23:37:14.172616 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:14.172580 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:37:23.776080 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:23.776042 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:37:24.237217 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.237171 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" podUID="f2cc7068-d367-48c6-a8b0-9c5edc639fa2" containerName="main" containerID="cri-o://4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4" gracePeriod=45 Jul 20 23:37:24.479334 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.479310 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:37:24.569947 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.569869 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tmp-dir\") pod \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " Jul 20 23:37:24.569947 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.569921 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qfg9\" (UniqueName: \"kubernetes.io/projected/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-kube-api-access-6qfg9\") pod \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " Jul 20 23:37:24.569947 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.569940 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-model-cache\") pod \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " Jul 20 23:37:24.570162 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570048 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-home\") pod \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " Jul 20 23:37:24.570162 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570093 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-dshm\") pod \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " Jul 20 23:37:24.570162 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570133 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tls-certs\") pod \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\" (UID: \"f2cc7068-d367-48c6-a8b0-9c5edc639fa2\") " Jul 20 23:37:24.570290 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570151 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2cc7068-d367-48c6-a8b0-9c5edc639fa2" (UID: "f2cc7068-d367-48c6-a8b0-9c5edc639fa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:24.570290 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570162 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2cc7068-d367-48c6-a8b0-9c5edc639fa2" (UID: "f2cc7068-d367-48c6-a8b0-9c5edc639fa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:24.570407 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570291 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-home" (OuterVolumeSpecName: "home") pod "f2cc7068-d367-48c6-a8b0-9c5edc639fa2" (UID: "f2cc7068-d367-48c6-a8b0-9c5edc639fa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:24.570466 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570414 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-model-cache\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:37:24.570466 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570430 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-home\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:37:24.570466 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.570443 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tmp-dir\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:37:24.572026 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.572000 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-kube-api-access-6qfg9" (OuterVolumeSpecName: "kube-api-access-6qfg9") pod "f2cc7068-d367-48c6-a8b0-9c5edc639fa2" (UID: "f2cc7068-d367-48c6-a8b0-9c5edc639fa2"). InnerVolumeSpecName "kube-api-access-6qfg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:37:24.572144 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.572129 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-dshm" (OuterVolumeSpecName: "dshm") pod "f2cc7068-d367-48c6-a8b0-9c5edc639fa2" (UID: "f2cc7068-d367-48c6-a8b0-9c5edc639fa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:37:24.572206 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.572140 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2cc7068-d367-48c6-a8b0-9c5edc639fa2" (UID: "f2cc7068-d367-48c6-a8b0-9c5edc639fa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:37:24.671801 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.671768 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qfg9\" (UniqueName: \"kubernetes.io/projected/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-kube-api-access-6qfg9\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:37:24.671801 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.671797 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-dshm\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:37:24.671801 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.671808 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cc7068-d367-48c6-a8b0-9c5edc639fa2-tls-certs\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:37:24.746351 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.746321 2574 generic.go:356] "Generic (PLEG): container finished" podID="f2cc7068-d367-48c6-a8b0-9c5edc639fa2" containerID="4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4" exitCode=0 Jul 20 23:37:24.746544 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.746358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" event={"ID":"f2cc7068-d367-48c6-a8b0-9c5edc639fa2","Type":"ContainerDied","Data":"4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4"} Jul 20 23:37:24.746544 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.746362 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" Jul 20 23:37:24.746544 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.746385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g" event={"ID":"f2cc7068-d367-48c6-a8b0-9c5edc639fa2","Type":"ContainerDied","Data":"36fd5b9c2ea889cdf96b2d1c1d1dbcbcdab376cd9f8ccaf06b2bd6d2b962ebc8"} Jul 20 23:37:24.746544 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.746401 2574 scope.go:117] "RemoveContainer" containerID="4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4" Jul 20 23:37:24.756155 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.756132 2574 scope.go:117] "RemoveContainer" containerID="4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4" Jul 20 23:37:24.756416 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:37:24.756395 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4\": container with ID starting with 4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4 not found: ID does not exist" containerID="4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4" Jul 20 23:37:24.756482 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.756429 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4"} err="failed to get container status \"4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4\": rpc error: code = NotFound desc = could not find container \"4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4\": container with ID starting with 4683d8ab88a1bb89d036a2e7ec01345422a2df33ef4e11da063e0a80c4680eb4 not found: ID does not exist" Jul 20 23:37:24.767717 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.767692 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g"] Jul 20 23:37:24.771139 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:24.771119 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/sub-test-model-5b9afe-kserve-69c5496855-z2h2g"] Jul 20 23:37:25.009344 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:25.009315 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cc7068-d367-48c6-a8b0-9c5edc639fa2" path="/var/lib/kubelet/pods/f2cc7068-d367-48c6-a8b0-9c5edc639fa2/volumes" Jul 20 23:37:31.343913 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:31.343883 2574 scope.go:117] "RemoveContainer" containerID="fcb510930726056f17110af6182f8188dc88bfb055a155cbe4ec9ed63f411aea" Jul 20 23:37:35.619464 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.619425 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7"] Jul 20 23:37:35.619851 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.619795 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cc7068-d367-48c6-a8b0-9c5edc639fa2" containerName="main" Jul 20 23:37:35.619851 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.619805 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cc7068-d367-48c6-a8b0-9c5edc639fa2" containerName="main" Jul 20 23:37:35.619928 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.619877 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cc7068-d367-48c6-a8b0-9c5edc639fa2" containerName="main" Jul 20 23:37:35.623055 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.623036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.626230 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.626196 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"default-dockercfg-v9wvl\"" Jul 20 23:37:35.626230 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.626213 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"openshift-service-ca.crt\"" Jul 20 23:37:35.626230 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.626223 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"kube-root-ca.crt\"" Jul 20 23:37:35.626480 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.626287 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5b9afe\"/\"rate-test-model-5b9afe-kserve-self-signed-certs\"" Jul 20 23:37:35.631163 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.631140 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7"] Jul 20 23:37:35.769772 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.769738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tls-certs\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.769963 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.769795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c64qw\" (UniqueName: \"kubernetes.io/projected/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-kube-api-access-c64qw\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.769963 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.769845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tmp-dir\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.769963 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.769874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-home\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.769963 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.769928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-model-cache\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.770132 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.769975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-dshm\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.870458 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-model-cache\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.870458 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-dshm\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.870458 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tls-certs\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.870744 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c64qw\" (UniqueName: \"kubernetes.io/projected/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-kube-api-access-c64qw\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.870744 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tmp-dir\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.870744 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-home\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.870906 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-model-cache\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.871000 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-home\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.871000 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.870980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tmp-dir\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.872873 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.872845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-dshm\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.873116 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.873101 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tls-certs\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.878224 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.878202 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c64qw\" (UniqueName: \"kubernetes.io/projected/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-kube-api-access-c64qw\") pod \"rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:35.932856 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:35.932819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:36.066982 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:36.066954 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7"] Jul 20 23:37:36.068413 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:37:36.068386 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fe84cb0_db31_4c6a_94d8_e3e3f8b0a4ae.slice/crio-726c22b58ec64bde38ede4da34c0c78d59bae5b90835df5ff0dc37d7cb5f459d WatchSource:0}: Error finding container 726c22b58ec64bde38ede4da34c0c78d59bae5b90835df5ff0dc37d7cb5f459d: Status 404 returned error can't find the container with id 726c22b58ec64bde38ede4da34c0c78d59bae5b90835df5ff0dc37d7cb5f459d Jul 20 23:37:36.809783 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:36.809745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" event={"ID":"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae","Type":"ContainerStarted","Data":"1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168"} Jul 20 23:37:36.810195 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:36.809792 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" event={"ID":"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae","Type":"ContainerStarted","Data":"726c22b58ec64bde38ede4da34c0c78d59bae5b90835df5ff0dc37d7cb5f459d"} Jul 20 23:37:39.271061 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:39.271008 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" podStartSLOduration=4.270992829 podStartE2EDuration="4.270992829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:37:36.826237734 +0000 UTC m=+2646.322692158" watchObservedRunningTime="2026-07-20 23:37:39.270992829 +0000 UTC m=+2648.767447252" Jul 20 23:37:39.272286 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:39.272268 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:37:45.933891 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:45.933860 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:45.934401 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:45.933911 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:45.946193 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:45.946162 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:46.873427 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:46.873400 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:37:55.593921 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:37:55.593887 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:38:10.676436 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:10.676401 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:38:18.699134 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:18.699100 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7"] Jul 20 23:38:19.615903 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.615874 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-139747668896304"] Jul 20 23:38:19.619376 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.619349 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896304" Jul 20 23:38:19.622292 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.622273 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-22ccc\"" Jul 20 23:38:19.625521 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.625484 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668896304"] Jul 20 23:38:19.726439 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.726407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjw6c\" (UniqueName: \"kubernetes.io/projected/059c9bfa-27fd-4508-b4f0-7c491e6e66ba-kube-api-access-bjw6c\") pod \"test-curl-7320-139747668896304\" (UID: \"059c9bfa-27fd-4508-b4f0-7c491e6e66ba\") " pod="opendatahub/test-curl-7320-139747668896304" Jul 20 23:38:19.827738 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.827708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjw6c\" (UniqueName: \"kubernetes.io/projected/059c9bfa-27fd-4508-b4f0-7c491e6e66ba-kube-api-access-bjw6c\") pod \"test-curl-7320-139747668896304\" (UID: \"059c9bfa-27fd-4508-b4f0-7c491e6e66ba\") " pod="opendatahub/test-curl-7320-139747668896304" Jul 20 23:38:19.836360 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.836329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjw6c\" (UniqueName: \"kubernetes.io/projected/059c9bfa-27fd-4508-b4f0-7c491e6e66ba-kube-api-access-bjw6c\") pod \"test-curl-7320-139747668896304\" (UID: \"059c9bfa-27fd-4508-b4f0-7c491e6e66ba\") " pod="opendatahub/test-curl-7320-139747668896304" Jul 20 23:38:19.930528 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:19.930438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896304" Jul 20 23:38:20.064874 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:20.064845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668896304"] Jul 20 23:38:20.066478 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:20.066444 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod059c9bfa_27fd_4508_b4f0_7c491e6e66ba.slice/crio-14c1408107b7a22118d63836f4d8e29e8e90bd4472667fda3c10e451f1c553af WatchSource:0}: Error finding container 14c1408107b7a22118d63836f4d8e29e8e90bd4472667fda3c10e451f1c553af: Status 404 returned error can't find the container with id 14c1408107b7a22118d63836f4d8e29e8e90bd4472667fda3c10e451f1c553af Jul 20 23:38:20.067795 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:20.067777 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 23:38:21.048610 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:21.048511 2574 generic.go:356] "Generic (PLEG): container finished" podID="059c9bfa-27fd-4508-b4f0-7c491e6e66ba" containerID="39c3fd25cd1c7b62a5c81bf7af79916ff36d276b0d2439ddf4f015a02ce0861b" exitCode=0 Jul 20 23:38:21.048610 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:21.048557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668896304" event={"ID":"059c9bfa-27fd-4508-b4f0-7c491e6e66ba","Type":"ContainerDied","Data":"39c3fd25cd1c7b62a5c81bf7af79916ff36d276b0d2439ddf4f015a02ce0861b"} Jul 20 23:38:21.048610 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:21.048581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668896304" event={"ID":"059c9bfa-27fd-4508-b4f0-7c491e6e66ba","Type":"ContainerStarted","Data":"14c1408107b7a22118d63836f4d8e29e8e90bd4472667fda3c10e451f1c553af"} Jul 20 23:38:22.186396 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.186373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896304" Jul 20 23:38:22.222212 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.222187 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7320-139747668896304_059c9bfa-27fd-4508-b4f0-7c491e6e66ba/test-curl-7320-139747668896304/0.log" Jul 20 23:38:22.245878 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.245845 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-139747668896304"] Jul 20 23:38:22.248023 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.248002 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-139747668896304"] Jul 20 23:38:22.250240 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.250215 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjw6c\" (UniqueName: \"kubernetes.io/projected/059c9bfa-27fd-4508-b4f0-7c491e6e66ba-kube-api-access-bjw6c\") pod \"059c9bfa-27fd-4508-b4f0-7c491e6e66ba\" (UID: \"059c9bfa-27fd-4508-b4f0-7c491e6e66ba\") " Jul 20 23:38:22.252333 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.252308 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059c9bfa-27fd-4508-b4f0-7c491e6e66ba-kube-api-access-bjw6c" (OuterVolumeSpecName: "kube-api-access-bjw6c") pod "059c9bfa-27fd-4508-b4f0-7c491e6e66ba" (UID: "059c9bfa-27fd-4508-b4f0-7c491e6e66ba"). InnerVolumeSpecName "kube-api-access-bjw6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:22.350837 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.350811 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjw6c\" (UniqueName: \"kubernetes.io/projected/059c9bfa-27fd-4508-b4f0-7c491e6e66ba-kube-api-access-bjw6c\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:22.359895 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.359873 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-139747668896432"] Jul 20 23:38:22.360214 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.360201 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="059c9bfa-27fd-4508-b4f0-7c491e6e66ba" containerName="test-curl-7320-139747668896304" Jul 20 23:38:22.360256 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.360216 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="059c9bfa-27fd-4508-b4f0-7c491e6e66ba" containerName="test-curl-7320-139747668896304" Jul 20 23:38:22.360291 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.360269 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="059c9bfa-27fd-4508-b4f0-7c491e6e66ba" containerName="test-curl-7320-139747668896304" Jul 20 23:38:22.363225 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.363210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:22.367538 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.367516 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668896432"] Jul 20 23:38:22.451875 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.451844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2h9f\" (UniqueName: \"kubernetes.io/projected/e275ffb2-2c3a-46db-b5ca-ab33e4affc09-kube-api-access-f2h9f\") pod \"test-curl-7320-139747668896432\" (UID: \"e275ffb2-2c3a-46db-b5ca-ab33e4affc09\") " pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:22.552724 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.552694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2h9f\" (UniqueName: \"kubernetes.io/projected/e275ffb2-2c3a-46db-b5ca-ab33e4affc09-kube-api-access-f2h9f\") pod \"test-curl-7320-139747668896432\" (UID: \"e275ffb2-2c3a-46db-b5ca-ab33e4affc09\") " pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:22.560630 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.560603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2h9f\" (UniqueName: \"kubernetes.io/projected/e275ffb2-2c3a-46db-b5ca-ab33e4affc09-kube-api-access-f2h9f\") pod \"test-curl-7320-139747668896432\" (UID: \"e275ffb2-2c3a-46db-b5ca-ab33e4affc09\") " pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:22.674939 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.674862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:22.806194 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:22.806123 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668896432"] Jul 20 23:38:22.808524 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:22.808479 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode275ffb2_2c3a_46db_b5ca_ab33e4affc09.slice/crio-427653ecf48dfb40f8f37dd8491b68ecc2ae9fad23e5925f4f15585a199a24b5 WatchSource:0}: Error finding container 427653ecf48dfb40f8f37dd8491b68ecc2ae9fad23e5925f4f15585a199a24b5: Status 404 returned error can't find the container with id 427653ecf48dfb40f8f37dd8491b68ecc2ae9fad23e5925f4f15585a199a24b5 Jul 20 23:38:23.010069 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:23.009969 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="059c9bfa-27fd-4508-b4f0-7c491e6e66ba" path="/var/lib/kubelet/pods/059c9bfa-27fd-4508-b4f0-7c491e6e66ba/volumes" Jul 20 23:38:23.059270 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:23.059237 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668896432" event={"ID":"e275ffb2-2c3a-46db-b5ca-ab33e4affc09","Type":"ContainerStarted","Data":"427653ecf48dfb40f8f37dd8491b68ecc2ae9fad23e5925f4f15585a199a24b5"} Jul 20 23:38:23.060397 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:23.060380 2574 scope.go:117] "RemoveContainer" containerID="39c3fd25cd1c7b62a5c81bf7af79916ff36d276b0d2439ddf4f015a02ce0861b" Jul 20 23:38:23.060540 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:23.060397 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896304" Jul 20 23:38:24.068708 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:24.068678 2574 generic.go:356] "Generic (PLEG): container finished" podID="e275ffb2-2c3a-46db-b5ca-ab33e4affc09" containerID="2f728dbade5032d3d0af6684b5626bff365c06843dd2565673419fd11f0d783f" exitCode=0 Jul 20 23:38:24.068708 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:24.068712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668896432" event={"ID":"e275ffb2-2c3a-46db-b5ca-ab33e4affc09","Type":"ContainerDied","Data":"2f728dbade5032d3d0af6684b5626bff365c06843dd2565673419fd11f0d783f"} Jul 20 23:38:25.210967 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.210936 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:25.230854 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.230827 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7320-139747668896432_e275ffb2-2c3a-46db-b5ca-ab33e4affc09/test-curl-7320-139747668896432/0.log" Jul 20 23:38:25.254031 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.254002 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-139747668896432"] Jul 20 23:38:25.259653 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.259631 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-139747668896432"] Jul 20 23:38:25.379102 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.379071 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2h9f\" (UniqueName: \"kubernetes.io/projected/e275ffb2-2c3a-46db-b5ca-ab33e4affc09-kube-api-access-f2h9f\") pod \"e275ffb2-2c3a-46db-b5ca-ab33e4affc09\" (UID: \"e275ffb2-2c3a-46db-b5ca-ab33e4affc09\") " Jul 20 23:38:25.381128 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.381103 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e275ffb2-2c3a-46db-b5ca-ab33e4affc09-kube-api-access-f2h9f" (OuterVolumeSpecName: "kube-api-access-f2h9f") pod "e275ffb2-2c3a-46db-b5ca-ab33e4affc09" (UID: "e275ffb2-2c3a-46db-b5ca-ab33e4affc09"). InnerVolumeSpecName "kube-api-access-f2h9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:25.420944 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.420910 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:25.421311 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.421293 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e275ffb2-2c3a-46db-b5ca-ab33e4affc09" containerName="test-curl-7320-139747668896432" Jul 20 23:38:25.421390 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.421313 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e275ffb2-2c3a-46db-b5ca-ab33e4affc09" containerName="test-curl-7320-139747668896432" Jul 20 23:38:25.421446 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.421436 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e275ffb2-2c3a-46db-b5ca-ab33e4affc09" containerName="test-curl-7320-139747668896432" Jul 20 23:38:25.424409 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.424389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:25.430907 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.430882 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:25.480163 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.480133 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2h9f\" (UniqueName: \"kubernetes.io/projected/e275ffb2-2c3a-46db-b5ca-ab33e4affc09-kube-api-access-f2h9f\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:25.571565 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.571529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:38:25.581278 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.581252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcb2q\" (UniqueName: \"kubernetes.io/projected/e809c8da-6c10-435f-b4e7-ed2287f823de-kube-api-access-gcb2q\") pod \"test-curl-7320-139747668918608\" (UID: \"e809c8da-6c10-435f-b4e7-ed2287f823de\") " pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:25.682134 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.682033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcb2q\" (UniqueName: \"kubernetes.io/projected/e809c8da-6c10-435f-b4e7-ed2287f823de-kube-api-access-gcb2q\") pod \"test-curl-7320-139747668918608\" (UID: \"e809c8da-6c10-435f-b4e7-ed2287f823de\") " pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:25.691025 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.690995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcb2q\" (UniqueName: \"kubernetes.io/projected/e809c8da-6c10-435f-b4e7-ed2287f823de-kube-api-access-gcb2q\") pod \"test-curl-7320-139747668918608\" (UID: \"e809c8da-6c10-435f-b4e7-ed2287f823de\") " pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:25.734592 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.734571 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:25.863294 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:25.863265 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:25.865184 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:25.865157 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode809c8da_6c10_435f_b4e7_ed2287f823de.slice/crio-04109a606ec4e7db14b56c78fd2317715e46fbd18cd5edfeb206804da0289fc1 WatchSource:0}: Error finding container 04109a606ec4e7db14b56c78fd2317715e46fbd18cd5edfeb206804da0289fc1: Status 404 returned error can't find the container with id 04109a606ec4e7db14b56c78fd2317715e46fbd18cd5edfeb206804da0289fc1 Jul 20 23:38:26.081684 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:26.081599 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:26.081876 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:26.081603 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="427653ecf48dfb40f8f37dd8491b68ecc2ae9fad23e5925f4f15585a199a24b5" Jul 20 23:38:26.083045 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:26.083018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668918608" event={"ID":"e809c8da-6c10-435f-b4e7-ed2287f823de","Type":"ContainerStarted","Data":"04109a606ec4e7db14b56c78fd2317715e46fbd18cd5edfeb206804da0289fc1"} Jul 20 23:38:26.092839 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:38:26.092813 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7320-139747668896432\" is forbidden: User \"system:node:ip-10-0-134-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-37.ec2.internal' and this object" podUID="e275ffb2-2c3a-46db-b5ca-ab33e4affc09" pod="opendatahub/test-curl-7320-139747668896432" Jul 20 23:38:27.009479 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:27.009441 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e275ffb2-2c3a-46db-b5ca-ab33e4affc09" path="/var/lib/kubelet/pods/e275ffb2-2c3a-46db-b5ca-ab33e4affc09/volumes" Jul 20 23:38:27.090068 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:27.090038 2574 generic.go:356] "Generic (PLEG): container finished" podID="e809c8da-6c10-435f-b4e7-ed2287f823de" containerID="2e455ca5d31aab8a89ed0a6622a90af01a5eaf7eb0c54c41965839e0a6688a1e" exitCode=0 Jul 20 23:38:27.090253 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:27.090125 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668918608" event={"ID":"e809c8da-6c10-435f-b4e7-ed2287f823de","Type":"ContainerDied","Data":"2e455ca5d31aab8a89ed0a6622a90af01a5eaf7eb0c54c41965839e0a6688a1e"} Jul 20 23:38:28.228485 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.228465 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:28.246422 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.246398 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7320-139747668918608_e809c8da-6c10-435f-b4e7-ed2287f823de/test-curl-7320-139747668918608/0.log" Jul 20 23:38:28.268939 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.268909 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:28.272328 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.272307 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:28.391771 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.391693 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:28.392081 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.392067 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e809c8da-6c10-435f-b4e7-ed2287f823de" containerName="test-curl-7320-139747668918608" Jul 20 23:38:28.392125 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.392083 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e809c8da-6c10-435f-b4e7-ed2287f823de" containerName="test-curl-7320-139747668918608" Jul 20 23:38:28.392165 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.392146 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e809c8da-6c10-435f-b4e7-ed2287f823de" containerName="test-curl-7320-139747668918608" Jul 20 23:38:28.395020 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.395005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:28.400808 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.400778 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:28.407797 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.407768 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcb2q\" (UniqueName: \"kubernetes.io/projected/e809c8da-6c10-435f-b4e7-ed2287f823de-kube-api-access-gcb2q\") pod \"e809c8da-6c10-435f-b4e7-ed2287f823de\" (UID: \"e809c8da-6c10-435f-b4e7-ed2287f823de\") " Jul 20 23:38:28.410216 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.410191 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e809c8da-6c10-435f-b4e7-ed2287f823de-kube-api-access-gcb2q" (OuterVolumeSpecName: "kube-api-access-gcb2q") pod "e809c8da-6c10-435f-b4e7-ed2287f823de" (UID: "e809c8da-6c10-435f-b4e7-ed2287f823de"). InnerVolumeSpecName "kube-api-access-gcb2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:28.509096 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.509059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lz4x\" (UniqueName: \"kubernetes.io/projected/ebf6280b-c59c-4efb-9b81-13c7bf818b99-kube-api-access-9lz4x\") pod \"test-curl-7320-139747668918608\" (UID: \"ebf6280b-c59c-4efb-9b81-13c7bf818b99\") " pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:28.509254 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.509106 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcb2q\" (UniqueName: \"kubernetes.io/projected/e809c8da-6c10-435f-b4e7-ed2287f823de-kube-api-access-gcb2q\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:28.609932 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.609902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lz4x\" (UniqueName: \"kubernetes.io/projected/ebf6280b-c59c-4efb-9b81-13c7bf818b99-kube-api-access-9lz4x\") pod \"test-curl-7320-139747668918608\" (UID: \"ebf6280b-c59c-4efb-9b81-13c7bf818b99\") " pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:28.618030 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.618006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lz4x\" (UniqueName: \"kubernetes.io/projected/ebf6280b-c59c-4efb-9b81-13c7bf818b99-kube-api-access-9lz4x\") pod \"test-curl-7320-139747668918608\" (UID: \"ebf6280b-c59c-4efb-9b81-13c7bf818b99\") " pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:28.710999 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.710925 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:28.842236 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:28.842211 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:28.843500 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:28.843459 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebf6280b_c59c_4efb_9b81_13c7bf818b99.slice/crio-ecdf83ea742baa9396718f42ea87e0d28862c964f137c2d3972ccb273ae6cc1b WatchSource:0}: Error finding container ecdf83ea742baa9396718f42ea87e0d28862c964f137c2d3972ccb273ae6cc1b: Status 404 returned error can't find the container with id ecdf83ea742baa9396718f42ea87e0d28862c964f137c2d3972ccb273ae6cc1b Jul 20 23:38:29.009481 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:29.009412 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e809c8da-6c10-435f-b4e7-ed2287f823de" path="/var/lib/kubelet/pods/e809c8da-6c10-435f-b4e7-ed2287f823de/volumes" Jul 20 23:38:29.101918 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:29.101895 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:29.102079 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:29.101939 2574 scope.go:117] "RemoveContainer" containerID="2e455ca5d31aab8a89ed0a6622a90af01a5eaf7eb0c54c41965839e0a6688a1e" Jul 20 23:38:29.103254 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:29.103224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668918608" event={"ID":"ebf6280b-c59c-4efb-9b81-13c7bf818b99","Type":"ContainerStarted","Data":"ecdf83ea742baa9396718f42ea87e0d28862c964f137c2d3972ccb273ae6cc1b"} Jul 20 23:38:30.109908 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:30.109877 2574 generic.go:356] "Generic (PLEG): container finished" podID="ebf6280b-c59c-4efb-9b81-13c7bf818b99" containerID="aac94ce7dd6e81c68d92ca6ce6006c3fbdd2f57ce158bee610e4a432a256a40f" exitCode=0 Jul 20 23:38:30.110313 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:30.109970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668918608" event={"ID":"ebf6280b-c59c-4efb-9b81-13c7bf818b99","Type":"ContainerDied","Data":"aac94ce7dd6e81c68d92ca6ce6006c3fbdd2f57ce158bee610e4a432a256a40f"} Jul 20 23:38:31.150542 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.150504 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:38:31.156995 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.156963 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:38:31.162901 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.162880 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:38:31.168958 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.168939 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sg4m5_6caa92d9-60c0-4340-8426-2f60704881bf/ovn-acl-logging/0.log" Jul 20 23:38:31.261060 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.261037 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:31.283650 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.283608 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7320-139747668918608_ebf6280b-c59c-4efb-9b81-13c7bf818b99/test-curl-7320-139747668918608/0.log" Jul 20 23:38:31.314709 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.314684 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:31.328770 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.325290 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-139747668918608"] Jul 20 23:38:31.368372 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.368312 2574 scope.go:117] "RemoveContainer" containerID="c0b92eacb8e8f896adaff233b0bbbbefad75aab2a801a3837058424e1df81e3e" Jul 20 23:38:31.432461 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.432434 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lz4x\" (UniqueName: \"kubernetes.io/projected/ebf6280b-c59c-4efb-9b81-13c7bf818b99-kube-api-access-9lz4x\") pod \"ebf6280b-c59c-4efb-9b81-13c7bf818b99\" (UID: \"ebf6280b-c59c-4efb-9b81-13c7bf818b99\") " Jul 20 23:38:31.434434 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.434410 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf6280b-c59c-4efb-9b81-13c7bf818b99-kube-api-access-9lz4x" (OuterVolumeSpecName: "kube-api-access-9lz4x") pod "ebf6280b-c59c-4efb-9b81-13c7bf818b99" (UID: "ebf6280b-c59c-4efb-9b81-13c7bf818b99"). InnerVolumeSpecName "kube-api-access-9lz4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:31.466597 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.466574 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:31.466949 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.466934 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf6280b-c59c-4efb-9b81-13c7bf818b99" containerName="test-curl-7320-139747668918608" Jul 20 23:38:31.466998 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.466950 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf6280b-c59c-4efb-9b81-13c7bf818b99" containerName="test-curl-7320-139747668918608" Jul 20 23:38:31.467032 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.467012 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf6280b-c59c-4efb-9b81-13c7bf818b99" containerName="test-curl-7320-139747668918608" Jul 20 23:38:31.471358 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.471343 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:31.484845 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.484824 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:31.533698 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.533668 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lz4x\" (UniqueName: \"kubernetes.io/projected/ebf6280b-c59c-4efb-9b81-13c7bf818b99-kube-api-access-9lz4x\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:31.634263 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.634192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gn6\" (UniqueName: \"kubernetes.io/projected/a422d930-be3d-44ce-9926-d470afb94404-kube-api-access-c4gn6\") pod \"test-curl-7320-139747668984864\" (UID: \"a422d930-be3d-44ce-9926-d470afb94404\") " pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:31.735450 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.735422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gn6\" (UniqueName: \"kubernetes.io/projected/a422d930-be3d-44ce-9926-d470afb94404-kube-api-access-c4gn6\") pod \"test-curl-7320-139747668984864\" (UID: \"a422d930-be3d-44ce-9926-d470afb94404\") " pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:31.744868 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.744843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gn6\" (UniqueName: \"kubernetes.io/projected/a422d930-be3d-44ce-9926-d470afb94404-kube-api-access-c4gn6\") pod \"test-curl-7320-139747668984864\" (UID: \"a422d930-be3d-44ce-9926-d470afb94404\") " pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:31.779625 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.779605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:31.907268 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:31.907243 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:31.909254 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:31.909228 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda422d930_be3d_44ce_9926_d470afb94404.slice/crio-2075dc6c26fbe209cbcf44e904da87f58de31c384ba0ac82361b6c97f72dc4ad WatchSource:0}: Error finding container 2075dc6c26fbe209cbcf44e904da87f58de31c384ba0ac82361b6c97f72dc4ad: Status 404 returned error can't find the container with id 2075dc6c26fbe209cbcf44e904da87f58de31c384ba0ac82361b6c97f72dc4ad Jul 20 23:38:32.123056 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:32.123026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668984864" event={"ID":"a422d930-be3d-44ce-9926-d470afb94404","Type":"ContainerStarted","Data":"2075dc6c26fbe209cbcf44e904da87f58de31c384ba0ac82361b6c97f72dc4ad"} Jul 20 23:38:32.124191 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:32.124175 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:32.124283 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:32.124180 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecdf83ea742baa9396718f42ea87e0d28862c964f137c2d3972ccb273ae6cc1b" Jul 20 23:38:32.134666 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:38:32.134638 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7320-139747668918608\" is forbidden: User \"system:node:ip-10-0-134-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-37.ec2.internal' and this object" podUID="ebf6280b-c59c-4efb-9b81-13c7bf818b99" pod="opendatahub/test-curl-7320-139747668918608" Jul 20 23:38:33.010687 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:33.010653 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf6280b-c59c-4efb-9b81-13c7bf818b99" path="/var/lib/kubelet/pods/ebf6280b-c59c-4efb-9b81-13c7bf818b99/volumes" Jul 20 23:38:33.130854 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:33.130816 2574 generic.go:356] "Generic (PLEG): container finished" podID="a422d930-be3d-44ce-9926-d470afb94404" containerID="1b0c6f1f789728a5e96a7ac5f5e649d0dd15b61bfe37400bdd3f660a0438f1a7" exitCode=0 Jul 20 23:38:33.131022 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:33.130898 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668984864" event={"ID":"a422d930-be3d-44ce-9926-d470afb94404","Type":"ContainerDied","Data":"1b0c6f1f789728a5e96a7ac5f5e649d0dd15b61bfe37400bdd3f660a0438f1a7"} Jul 20 23:38:33.719705 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:33.719666 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" podUID="2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" containerName="main" containerID="cri-o://1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168" gracePeriod=45 Jul 20 23:38:33.975849 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:33.975789 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:38:34.138066 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.138029 2574 generic.go:356] "Generic (PLEG): container finished" podID="2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" containerID="1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168" exitCode=0 Jul 20 23:38:34.138066 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.138071 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" Jul 20 23:38:34.138554 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.138116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" event={"ID":"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae","Type":"ContainerDied","Data":"1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168"} Jul 20 23:38:34.138554 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.138166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7" event={"ID":"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae","Type":"ContainerDied","Data":"726c22b58ec64bde38ede4da34c0c78d59bae5b90835df5ff0dc37d7cb5f459d"} Jul 20 23:38:34.138554 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.138197 2574 scope.go:117] "RemoveContainer" containerID="1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168" Jul 20 23:38:34.146973 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.146948 2574 scope.go:117] "RemoveContainer" containerID="1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168" Jul 20 23:38:34.147232 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:38:34.147213 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168\": container with ID starting with 1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168 not found: ID does not exist" containerID="1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168" Jul 20 23:38:34.147306 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.147246 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168"} err="failed to get container status \"1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168\": rpc error: code = NotFound desc = could not find container \"1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168\": container with ID starting with 1aac03d4ea6d4cd9e0f92d7593e18f3ae24266315e5d2d29b2e886ddfed75168 not found: ID does not exist" Jul 20 23:38:34.157568 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.157546 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-dshm\") pod \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " Jul 20 23:38:34.157692 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.157587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-home\") pod \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " Jul 20 23:38:34.157692 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.157625 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tls-certs\") pod \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " Jul 20 23:38:34.157692 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.157659 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tmp-dir\") pod \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " Jul 20 23:38:34.157868 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.157738 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c64qw\" (UniqueName: \"kubernetes.io/projected/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-kube-api-access-c64qw\") pod \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " Jul 20 23:38:34.157868 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.157762 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-model-cache\") pod \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\" (UID: \"2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae\") " Jul 20 23:38:34.157976 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.157914 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-home" (OuterVolumeSpecName: "home") pod "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" (UID: "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:34.158062 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.158041 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-home\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:34.158124 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.158063 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" (UID: "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:34.158170 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.158122 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" (UID: "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:34.159614 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.159594 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-dshm" (OuterVolumeSpecName: "dshm") pod "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" (UID: "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 23:38:34.160148 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.160129 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" (UID: "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 23:38:34.160203 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.160134 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-kube-api-access-c64qw" (OuterVolumeSpecName: "kube-api-access-c64qw") pod "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" (UID: "2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae"). InnerVolumeSpecName "kube-api-access-c64qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:34.258476 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.258449 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-dshm\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:34.258476 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.258475 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tls-certs\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:34.258667 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.258484 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-tmp-dir\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:34.258667 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.258507 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c64qw\" (UniqueName: \"kubernetes.io/projected/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-kube-api-access-c64qw\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:34.258667 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.258516 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae-model-cache\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:34.262798 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.262775 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:34.281572 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.281541 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7320-139747668984864_a422d930-be3d-44ce-9926-d470afb94404/test-curl-7320-139747668984864/0.log" Jul 20 23:38:34.305884 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.305856 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:34.311586 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.311559 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:34.359645 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.359620 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4gn6\" (UniqueName: \"kubernetes.io/projected/a422d930-be3d-44ce-9926-d470afb94404-kube-api-access-c4gn6\") pod \"a422d930-be3d-44ce-9926-d470afb94404\" (UID: \"a422d930-be3d-44ce-9926-d470afb94404\") " Jul 20 23:38:34.361679 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.361659 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a422d930-be3d-44ce-9926-d470afb94404-kube-api-access-c4gn6" (OuterVolumeSpecName: "kube-api-access-c4gn6") pod "a422d930-be3d-44ce-9926-d470afb94404" (UID: "a422d930-be3d-44ce-9926-d470afb94404"). InnerVolumeSpecName "kube-api-access-c4gn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:34.448690 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.448666 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:34.448998 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.448987 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" containerName="main" Jul 20 23:38:34.449038 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.449000 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" containerName="main" Jul 20 23:38:34.449038 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.449010 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a422d930-be3d-44ce-9926-d470afb94404" containerName="test-curl-7320-139747668984864" Jul 20 23:38:34.449038 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.449017 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a422d930-be3d-44ce-9926-d470afb94404" containerName="test-curl-7320-139747668984864" Jul 20 23:38:34.449129 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.449072 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a422d930-be3d-44ce-9926-d470afb94404" containerName="test-curl-7320-139747668984864" Jul 20 23:38:34.449129 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.449080 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" containerName="main" Jul 20 23:38:34.452039 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.452022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:34.458049 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.458028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:34.460395 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.460370 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gn6\" (UniqueName: \"kubernetes.io/projected/a422d930-be3d-44ce-9926-d470afb94404-kube-api-access-c4gn6\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:34.461720 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.461696 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7"] Jul 20 23:38:34.466190 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.466170 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5b9afe/rate-test-model-5b9afe-kserve-799dd54f4c-zpbg7"] Jul 20 23:38:34.561586 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.561511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9w7\" (UniqueName: \"kubernetes.io/projected/ef6ef1cb-4d48-4b5a-8fc2-3952c026a318-kube-api-access-sx9w7\") pod \"test-curl-7320-139747668984864\" (UID: \"ef6ef1cb-4d48-4b5a-8fc2-3952c026a318\") " pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:34.662322 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.662290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9w7\" (UniqueName: \"kubernetes.io/projected/ef6ef1cb-4d48-4b5a-8fc2-3952c026a318-kube-api-access-sx9w7\") pod \"test-curl-7320-139747668984864\" (UID: \"ef6ef1cb-4d48-4b5a-8fc2-3952c026a318\") " pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:34.672811 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.672787 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9w7\" (UniqueName: \"kubernetes.io/projected/ef6ef1cb-4d48-4b5a-8fc2-3952c026a318-kube-api-access-sx9w7\") pod \"test-curl-7320-139747668984864\" (UID: \"ef6ef1cb-4d48-4b5a-8fc2-3952c026a318\") " pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:34.762397 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.762366 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:34.891291 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:34.891257 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:34.893090 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:34.893067 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef6ef1cb_4d48_4b5a_8fc2_3952c026a318.slice/crio-5aa7f2213c5a009d9be2f9dd7d33c114942a116dc7d29c60c8eb0998cc81fd46 WatchSource:0}: Error finding container 5aa7f2213c5a009d9be2f9dd7d33c114942a116dc7d29c60c8eb0998cc81fd46: Status 404 returned error can't find the container with id 5aa7f2213c5a009d9be2f9dd7d33c114942a116dc7d29c60c8eb0998cc81fd46 Jul 20 23:38:35.009134 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:35.009101 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae" path="/var/lib/kubelet/pods/2fe84cb0-db31-4c6a-94d8-e3e3f8b0a4ae/volumes" Jul 20 23:38:35.009441 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:35.009429 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a422d930-be3d-44ce-9926-d470afb94404" path="/var/lib/kubelet/pods/a422d930-be3d-44ce-9926-d470afb94404/volumes" Jul 20 23:38:35.144334 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:35.144294 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668984864" event={"ID":"ef6ef1cb-4d48-4b5a-8fc2-3952c026a318","Type":"ContainerStarted","Data":"5aa7f2213c5a009d9be2f9dd7d33c114942a116dc7d29c60c8eb0998cc81fd46"} Jul 20 23:38:35.145524 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:35.145486 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:35.145642 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:35.145528 2574 scope.go:117] "RemoveContainer" containerID="1b0c6f1f789728a5e96a7ac5f5e649d0dd15b61bfe37400bdd3f660a0438f1a7" Jul 20 23:38:36.153420 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:36.153385 2574 generic.go:356] "Generic (PLEG): container finished" podID="ef6ef1cb-4d48-4b5a-8fc2-3952c026a318" containerID="f323d86c8b1d25952cd813a51c29e5128934ae64eb95413097ee802f85c09d45" exitCode=0 Jul 20 23:38:36.153854 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:36.153466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-139747668984864" event={"ID":"ef6ef1cb-4d48-4b5a-8fc2-3952c026a318","Type":"ContainerDied","Data":"f323d86c8b1d25952cd813a51c29e5128934ae64eb95413097ee802f85c09d45"} Jul 20 23:38:37.297287 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.297259 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:37.322278 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.322253 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7320-139747668984864_ef6ef1cb-4d48-4b5a-8fc2-3952c026a318/test-curl-7320-139747668984864/0.log" Jul 20 23:38:37.344196 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.344162 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:37.348394 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.348373 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-139747668984864"] Jul 20 23:38:37.486257 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.486184 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx9w7\" (UniqueName: \"kubernetes.io/projected/ef6ef1cb-4d48-4b5a-8fc2-3952c026a318-kube-api-access-sx9w7\") pod \"ef6ef1cb-4d48-4b5a-8fc2-3952c026a318\" (UID: \"ef6ef1cb-4d48-4b5a-8fc2-3952c026a318\") " Jul 20 23:38:37.488198 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.488170 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6ef1cb-4d48-4b5a-8fc2-3952c026a318-kube-api-access-sx9w7" (OuterVolumeSpecName: "kube-api-access-sx9w7") pod "ef6ef1cb-4d48-4b5a-8fc2-3952c026a318" (UID: "ef6ef1cb-4d48-4b5a-8fc2-3952c026a318"). InnerVolumeSpecName "kube-api-access-sx9w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 23:38:37.546474 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.546448 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk"] Jul 20 23:38:37.546857 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.546842 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6ef1cb-4d48-4b5a-8fc2-3952c026a318" containerName="test-curl-7320-139747668984864" Jul 20 23:38:37.546923 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.546861 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6ef1cb-4d48-4b5a-8fc2-3952c026a318" containerName="test-curl-7320-139747668984864" Jul 20 23:38:37.546974 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.546934 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6ef1cb-4d48-4b5a-8fc2-3952c026a318" containerName="test-curl-7320-139747668984864" Jul 20 23:38:37.549917 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.549902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" Jul 20 23:38:37.552019 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.551999 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:38:37.552100 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.552066 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:38:37.557031 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.557009 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk"] Jul 20 23:38:37.587624 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.587600 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9w7\" (UniqueName: \"kubernetes.io/projected/ef6ef1cb-4d48-4b5a-8fc2-3952c026a318-kube-api-access-sx9w7\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:37.688152 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.688121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d0fb37-c854-42ae-a434-b93b260ee2b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk\" (UID: \"26d0fb37-c854-42ae-a434-b93b260ee2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" Jul 20 23:38:37.789360 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.789278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d0fb37-c854-42ae-a434-b93b260ee2b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk\" (UID: \"26d0fb37-c854-42ae-a434-b93b260ee2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" Jul 20 23:38:37.789952 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.789932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d0fb37-c854-42ae-a434-b93b260ee2b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk\" (UID: \"26d0fb37-c854-42ae-a434-b93b260ee2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" Jul 20 23:38:37.861200 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.861173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" Jul 20 23:38:37.993124 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:37.993100 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk"] Jul 20 23:38:37.994400 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:37.994370 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d0fb37_c854_42ae_a434_b93b260ee2b4.slice/crio-19bcabe4298e06c176b859e7c3a47d4edab6606867221500736ee81cedf2f72e WatchSource:0}: Error finding container 19bcabe4298e06c176b859e7c3a47d4edab6606867221500736ee81cedf2f72e: Status 404 returned error can't find the container with id 19bcabe4298e06c176b859e7c3a47d4edab6606867221500736ee81cedf2f72e Jul 20 23:38:38.166269 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:38.166240 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" event={"ID":"26d0fb37-c854-42ae-a434-b93b260ee2b4","Type":"ContainerStarted","Data":"19bcabe4298e06c176b859e7c3a47d4edab6606867221500736ee81cedf2f72e"} Jul 20 23:38:38.167395 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:38.167377 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa7f2213c5a009d9be2f9dd7d33c114942a116dc7d29c60c8eb0998cc81fd46" Jul 20 23:38:38.167395 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:38.167393 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:38.177270 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:38:38.177246 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7320-139747668984864\" is forbidden: User \"system:node:ip-10-0-134-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-37.ec2.internal' and this object" podUID="ef6ef1cb-4d48-4b5a-8fc2-3952c026a318" pod="opendatahub/test-curl-7320-139747668984864" Jul 20 23:38:38.654516 ip-10-0-134-37 kubenswrapper[2574]: E0720 23:38:38.654464 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d0fb37_c854_42ae_a434_b93b260ee2b4.slice/crio-conmon-6624b39544eed906379ddbf3e547da9318398c605d9efb4673dcef96091ba840.scope\": RecentStats: unable to find data in memory cache]" Jul 20 23:38:39.009473 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:39.009430 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef6ef1cb-4d48-4b5a-8fc2-3952c026a318" path="/var/lib/kubelet/pods/ef6ef1cb-4d48-4b5a-8fc2-3952c026a318/volumes" Jul 20 23:38:39.173789 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:39.173752 2574 generic.go:356] "Generic (PLEG): container finished" podID="26d0fb37-c854-42ae-a434-b93b260ee2b4" containerID="6624b39544eed906379ddbf3e547da9318398c605d9efb4673dcef96091ba840" exitCode=0 Jul 20 23:38:39.173960 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:39.173813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" event={"ID":"26d0fb37-c854-42ae-a434-b93b260ee2b4","Type":"ContainerDied","Data":"6624b39544eed906379ddbf3e547da9318398c605d9efb4673dcef96091ba840"} Jul 20 23:38:40.313539 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:40.313516 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" Jul 20 23:38:40.409781 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:40.409750 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d0fb37-c854-42ae-a434-b93b260ee2b4-maas-api-service-ca\") pod \"26d0fb37-c854-42ae-a434-b93b260ee2b4\" (UID: \"26d0fb37-c854-42ae-a434-b93b260ee2b4\") " Jul 20 23:38:40.410112 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:40.410088 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d0fb37-c854-42ae-a434-b93b260ee2b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26d0fb37-c854-42ae-a434-b93b260ee2b4" (UID: "26d0fb37-c854-42ae-a434-b93b260ee2b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:38:40.510422 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:40.510358 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d0fb37-c854-42ae-a434-b93b260ee2b4-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:38:41.186829 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:41.186802 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" Jul 20 23:38:41.186829 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:41.186815 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk" event={"ID":"26d0fb37-c854-42ae-a434-b93b260ee2b4","Type":"ContainerDied","Data":"19bcabe4298e06c176b859e7c3a47d4edab6606867221500736ee81cedf2f72e"} Jul 20 23:38:41.187030 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:41.186870 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19bcabe4298e06c176b859e7c3a47d4edab6606867221500736ee81cedf2f72e" Jul 20 23:38:51.067639 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:51.067602 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:38:52.626203 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:52.626164 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk"] Jul 20 23:38:52.630761 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:52.630735 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-107ae7-p2mqk"] Jul 20 23:38:53.009015 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:53.008935 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d0fb37-c854-42ae-a434-b93b260ee2b4" path="/var/lib/kubelet/pods/26d0fb37-c854-42ae-a434-b93b260ee2b4/volumes" Jul 20 23:38:59.333994 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.333947 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn"] Jul 20 23:38:59.334483 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.334321 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d0fb37-c854-42ae-a434-b93b260ee2b4" containerName="revoke-keys" Jul 20 23:38:59.334483 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.334333 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d0fb37-c854-42ae-a434-b93b260ee2b4" containerName="revoke-keys" Jul 20 23:38:59.334483 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.334398 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d0fb37-c854-42ae-a434-b93b260ee2b4" containerName="revoke-keys" Jul 20 23:38:59.339964 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.339944 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" Jul 20 23:38:59.342850 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.342828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgvh2\"" Jul 20 23:38:59.343017 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.342998 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 23:38:59.350212 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.350192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn"] Jul 20 23:38:59.455353 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.455318 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0a546d-c7da-46c0-b79e-877fa7d20116-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn\" (UID: \"da0a546d-c7da-46c0-b79e-877fa7d20116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" Jul 20 23:38:59.556014 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.555977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0a546d-c7da-46c0-b79e-877fa7d20116-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn\" (UID: \"da0a546d-c7da-46c0-b79e-877fa7d20116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" Jul 20 23:38:59.556620 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.556601 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0a546d-c7da-46c0-b79e-877fa7d20116-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn\" (UID: \"da0a546d-c7da-46c0-b79e-877fa7d20116\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" Jul 20 23:38:59.650741 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.650689 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" Jul 20 23:38:59.779921 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:38:59.779894 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn"] Jul 20 23:38:59.782659 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:38:59.782631 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0a546d_c7da_46c0_b79e_877fa7d20116.slice/crio-bc93cc73038619275d28f3e9aee53bea380961ee224f76a8abebf4f0702f82e5 WatchSource:0}: Error finding container bc93cc73038619275d28f3e9aee53bea380961ee224f76a8abebf4f0702f82e5: Status 404 returned error can't find the container with id bc93cc73038619275d28f3e9aee53bea380961ee224f76a8abebf4f0702f82e5 Jul 20 23:39:00.292445 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:00.292358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" event={"ID":"da0a546d-c7da-46c0-b79e-877fa7d20116","Type":"ContainerStarted","Data":"47a6955aa5e55bb20377a6c15c08ef61a5c30c4fcfb8e2a6e9807ed83e349794"} Jul 20 23:39:00.292445 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:00.292416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" event={"ID":"da0a546d-c7da-46c0-b79e-877fa7d20116","Type":"ContainerStarted","Data":"bc93cc73038619275d28f3e9aee53bea380961ee224f76a8abebf4f0702f82e5"} Jul 20 23:39:00.307972 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:00.307925 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" podStartSLOduration=1.30791228 podStartE2EDuration="1.30791228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:39:00.306191339 +0000 UTC m=+2729.802645765" watchObservedRunningTime="2026-07-20 23:39:00.30791228 +0000 UTC m=+2729.804366751" Jul 20 23:39:01.298941 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:01.298910 2574 generic.go:356] "Generic (PLEG): container finished" podID="da0a546d-c7da-46c0-b79e-877fa7d20116" containerID="47a6955aa5e55bb20377a6c15c08ef61a5c30c4fcfb8e2a6e9807ed83e349794" exitCode=0 Jul 20 23:39:01.299365 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:01.298967 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" event={"ID":"da0a546d-c7da-46c0-b79e-877fa7d20116","Type":"ContainerDied","Data":"47a6955aa5e55bb20377a6c15c08ef61a5c30c4fcfb8e2a6e9807ed83e349794"} Jul 20 23:39:02.432333 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:02.432309 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" Jul 20 23:39:02.483453 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:02.483429 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0a546d-c7da-46c0-b79e-877fa7d20116-maas-api-service-ca\") pod \"da0a546d-c7da-46c0-b79e-877fa7d20116\" (UID: \"da0a546d-c7da-46c0-b79e-877fa7d20116\") " Jul 20 23:39:02.483771 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:02.483751 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0a546d-c7da-46c0-b79e-877fa7d20116-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da0a546d-c7da-46c0-b79e-877fa7d20116" (UID: "da0a546d-c7da-46c0-b79e-877fa7d20116"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 23:39:02.584940 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:02.584861 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0a546d-c7da-46c0-b79e-877fa7d20116-maas-api-service-ca\") on node \"ip-10-0-134-37.ec2.internal\" DevicePath \"\"" Jul 20 23:39:03.310914 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:03.310823 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" Jul 20 23:39:03.310914 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:03.310838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn" event={"ID":"da0a546d-c7da-46c0-b79e-877fa7d20116","Type":"ContainerDied","Data":"bc93cc73038619275d28f3e9aee53bea380961ee224f76a8abebf4f0702f82e5"} Jul 20 23:39:03.310914 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:03.310891 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc93cc73038619275d28f3e9aee53bea380961ee224f76a8abebf4f0702f82e5" Jul 20 23:39:12.674619 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:12.674581 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gnxpv"] Jul 20 23:39:14.452904 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:14.452871 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn"] Jul 20 23:39:14.455599 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:14.455578 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5b9afe-xtbvn"] Jul 20 23:39:15.008626 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:15.008589 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da0a546d-c7da-46c0-b79e-877fa7d20116" path="/var/lib/kubelet/pods/da0a546d-c7da-46c0-b79e-877fa7d20116/volumes" Jul 20 23:39:27.295175 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:27.295138 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vt2g2_8cb9248c-af87-4f14-82f8-e2be59923b56/manager/0.log" Jul 20 23:39:27.408168 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:27.408138 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-ccxxt_f125e432-a510-404a-9039-e45602ab288b/copy-manifests/0.log" Jul 20 23:39:27.414761 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:27.414737 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-ccxxt_f125e432-a510-404a-9039-e45602ab288b/manager/0.log" Jul 20 23:39:28.115903 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:28.115872 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29743155-mhjsg_d44d203b-5f19-4b2e-a6a1-4b2dca917483/cleanup/0.log" Jul 20 23:39:28.218975 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:28.218941 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29743170-hlngt_0a29b6a3-9213-4fcf-b721-a10f10822d94/cleanup/0.log" Jul 20 23:39:29.068092 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.068060 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk_46361911-bbbf-4020-9faa-fb1855680140/util/0.log" Jul 20 23:39:29.073947 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.073921 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk_46361911-bbbf-4020-9faa-fb1855680140/pull/0.log" Jul 20 23:39:29.079665 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.079645 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk_46361911-bbbf-4020-9faa-fb1855680140/extract/0.log" Jul 20 23:39:29.193200 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.193172 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph_ba50946f-3dcf-456a-a79c-26e12a195f31/util/0.log" Jul 20 23:39:29.198751 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.198729 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph_ba50946f-3dcf-456a-a79c-26e12a195f31/pull/0.log" Jul 20 23:39:29.204332 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.204313 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph_ba50946f-3dcf-456a-a79c-26e12a195f31/extract/0.log" Jul 20 23:39:29.313961 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.313931 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55_169de9a2-173a-422f-82ae-4f5622517f68/util/0.log" Jul 20 23:39:29.320324 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.320253 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55_169de9a2-173a-422f-82ae-4f5622517f68/pull/0.log" Jul 20 23:39:29.325987 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.325963 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55_169de9a2-173a-422f-82ae-4f5622517f68/extract/0.log" Jul 20 23:39:29.438660 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.438629 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z_90b470dc-df45-4d81-b1b3-b0b4b05c5156/util/0.log" Jul 20 23:39:29.445010 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.444990 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z_90b470dc-df45-4d81-b1b3-b0b4b05c5156/pull/0.log" Jul 20 23:39:29.450701 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.450679 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z_90b470dc-df45-4d81-b1b3-b0b4b05c5156/extract/0.log" Jul 20 23:39:29.785053 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:29.785026 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gh859_457b7481-e556-4d43-aa53-19c2da729772/manager/0.log" Jul 20 23:39:30.230852 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:30.230825 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gnxpv_2d94b952-f5bc-4653-b2b6-5b7add4dd7a7/limitador/0.log" Jul 20 23:39:31.017873 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:31.017840 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-pnlcd_81a76f81-b28a-4956-abff-92d4a02a2a0c/kube-auth-proxy/0.log" Jul 20 23:39:32.010688 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:32.010625 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rf4kq_ad57446a-fac7-4c29-ac96-f3137a0d849e/main/0.log" Jul 20 23:39:32.340234 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:32.340206 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4h6vx_86978c0c-7fb3-4b59-b07e-80dc29b110ce/main/0.log" Jul 20 23:39:38.108354 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.108319 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95lz9/must-gather-dbd5m"] Jul 20 23:39:38.110808 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.108709 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da0a546d-c7da-46c0-b79e-877fa7d20116" containerName="revoke-keys" Jul 20 23:39:38.110808 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.108720 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="da0a546d-c7da-46c0-b79e-877fa7d20116" containerName="revoke-keys" Jul 20 23:39:38.110808 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.108782 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="da0a546d-c7da-46c0-b79e-877fa7d20116" containerName="revoke-keys" Jul 20 23:39:38.112038 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.112020 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.115519 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.115477 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-95lz9\"/\"default-dockercfg-rh8pd\"" Jul 20 23:39:38.115681 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.115531 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95lz9\"/\"openshift-service-ca.crt\"" Jul 20 23:39:38.116387 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.116370 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-95lz9\"/\"kube-root-ca.crt\"" Jul 20 23:39:38.128363 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.128331 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95lz9/must-gather-dbd5m"] Jul 20 23:39:38.202382 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.202347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7td\" (UniqueName: \"kubernetes.io/projected/d801f8f1-0352-4180-8da8-ef2c914c2f4f-kube-api-access-tg7td\") pod \"must-gather-dbd5m\" (UID: \"d801f8f1-0352-4180-8da8-ef2c914c2f4f\") " pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.202382 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.202383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d801f8f1-0352-4180-8da8-ef2c914c2f4f-must-gather-output\") pod \"must-gather-dbd5m\" (UID: \"d801f8f1-0352-4180-8da8-ef2c914c2f4f\") " pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.303325 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.303284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7td\" (UniqueName: \"kubernetes.io/projected/d801f8f1-0352-4180-8da8-ef2c914c2f4f-kube-api-access-tg7td\") pod \"must-gather-dbd5m\" (UID: \"d801f8f1-0352-4180-8da8-ef2c914c2f4f\") " pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.303539 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.303356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d801f8f1-0352-4180-8da8-ef2c914c2f4f-must-gather-output\") pod \"must-gather-dbd5m\" (UID: \"d801f8f1-0352-4180-8da8-ef2c914c2f4f\") " pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.303751 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.303733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d801f8f1-0352-4180-8da8-ef2c914c2f4f-must-gather-output\") pod \"must-gather-dbd5m\" (UID: \"d801f8f1-0352-4180-8da8-ef2c914c2f4f\") " pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.311281 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.311260 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7td\" (UniqueName: \"kubernetes.io/projected/d801f8f1-0352-4180-8da8-ef2c914c2f4f-kube-api-access-tg7td\") pod \"must-gather-dbd5m\" (UID: \"d801f8f1-0352-4180-8da8-ef2c914c2f4f\") " pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.422083 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.421995 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95lz9/must-gather-dbd5m" Jul 20 23:39:38.560508 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:38.560463 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95lz9/must-gather-dbd5m"] Jul 20 23:39:38.562708 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:39:38.562678 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd801f8f1_0352_4180_8da8_ef2c914c2f4f.slice/crio-37cc906c59beef4fe47fa3e5e5e82d390b603c1983acee1f7f0b04d1f0b78f04 WatchSource:0}: Error finding container 37cc906c59beef4fe47fa3e5e5e82d390b603c1983acee1f7f0b04d1f0b78f04: Status 404 returned error can't find the container with id 37cc906c59beef4fe47fa3e5e5e82d390b603c1983acee1f7f0b04d1f0b78f04 Jul 20 23:39:39.511482 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:39.511445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95lz9/must-gather-dbd5m" event={"ID":"d801f8f1-0352-4180-8da8-ef2c914c2f4f","Type":"ContainerStarted","Data":"37cc906c59beef4fe47fa3e5e5e82d390b603c1983acee1f7f0b04d1f0b78f04"} Jul 20 23:39:40.521148 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:40.521110 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95lz9/must-gather-dbd5m" event={"ID":"d801f8f1-0352-4180-8da8-ef2c914c2f4f","Type":"ContainerStarted","Data":"9e820d04d6beb4152dac2a32c54ebb6c9c139f2e7aa2d145d346dd82896e84ef"} Jul 20 23:39:40.521566 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:40.521159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95lz9/must-gather-dbd5m" event={"ID":"d801f8f1-0352-4180-8da8-ef2c914c2f4f","Type":"ContainerStarted","Data":"727b7d3a58d495123a6255aa91a88169312369da5d32a8c267ff2e8de5d85e22"} Jul 20 23:39:40.537605 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:40.537551 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95lz9/must-gather-dbd5m" podStartSLOduration=1.627567987 podStartE2EDuration="2.537533957s" totalImagesPullingTime="909.96597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 23:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:39:40.534862097 +0000 UTC m=+2770.031316539" watchObservedRunningTime="2026-07-20 23:39:40.537533957 +0000 UTC m=+2770.033988380" Jul 20 23:39:41.115175 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:41.115145 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xl88f_ca14ac6b-9250-4bfc-84b6-ceb7b92f7b39/global-pull-secret-syncer/0.log" Jul 20 23:39:41.234828 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:41.234767 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nnwwg_4d5f7832-f567-42b5-966f-4084d961c60b/konnectivity-agent/0.log" Jul 20 23:39:41.306002 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:41.305973 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-37.ec2.internal_40387f4cb1b49043baacd9f958633c93/haproxy/0.log" Jul 20 23:39:46.136150 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.136121 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk_46361911-bbbf-4020-9faa-fb1855680140/extract/0.log" Jul 20 23:39:46.167934 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.167907 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk_46361911-bbbf-4020-9faa-fb1855680140/util/0.log" Jul 20 23:39:46.193212 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.193182 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759b97gk_46361911-bbbf-4020-9faa-fb1855680140/pull/0.log" Jul 20 23:39:46.221138 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.221110 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph_ba50946f-3dcf-456a-a79c-26e12a195f31/extract/0.log" Jul 20 23:39:46.242905 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.242875 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph_ba50946f-3dcf-456a-a79c-26e12a195f31/util/0.log" Jul 20 23:39:46.264839 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.264772 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mtdph_ba50946f-3dcf-456a-a79c-26e12a195f31/pull/0.log" Jul 20 23:39:46.292448 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.292385 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55_169de9a2-173a-422f-82ae-4f5622517f68/extract/0.log" Jul 20 23:39:46.316484 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.316448 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55_169de9a2-173a-422f-82ae-4f5622517f68/util/0.log" Jul 20 23:39:46.348366 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.348293 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738vm55_169de9a2-173a-422f-82ae-4f5622517f68/pull/0.log" Jul 20 23:39:46.379583 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.379550 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z_90b470dc-df45-4d81-b1b3-b0b4b05c5156/extract/0.log" Jul 20 23:39:46.401048 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.400953 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z_90b470dc-df45-4d81-b1b3-b0b4b05c5156/util/0.log" Jul 20 23:39:46.423089 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.423056 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fbs4z_90b470dc-df45-4d81-b1b3-b0b4b05c5156/pull/0.log" Jul 20 23:39:46.828238 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:46.828160 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-gh859_457b7481-e556-4d43-aa53-19c2da729772/manager/0.log" Jul 20 23:39:47.062366 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:47.062154 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gnxpv_2d94b952-f5bc-4653-b2b6-5b7add4dd7a7/limitador/0.log" Jul 20 23:39:49.214483 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:49.214448 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rstkt_fef0358d-b9a5-4a60-a522-01cf7860103a/node-exporter/0.log" Jul 20 23:39:49.238005 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:49.237978 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rstkt_fef0358d-b9a5-4a60-a522-01cf7860103a/kube-rbac-proxy/0.log" Jul 20 23:39:49.259128 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:49.259096 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rstkt_fef0358d-b9a5-4a60-a522-01cf7860103a/init-textfile/0.log" Jul 20 23:39:51.058439 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.058403 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-tgjs4_2e4c6d2b-cce8-4b70-af8d-8791605ee28e/networking-console-plugin/0.log" Jul 20 23:39:51.071983 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.071942 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6"] Jul 20 23:39:51.080308 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.080271 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.083965 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.083933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6"] Jul 20 23:39:51.132814 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.131912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-kube-api-access-ldcpm\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.132814 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.131995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-podres\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.132814 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.132028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-proc\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.132814 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.132072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-sys\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.132814 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.132119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-lib-modules\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.233314 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.233261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-kube-api-access-ldcpm\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.233693 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.233670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-podres\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.233841 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.233828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-proc\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.233975 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.233962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-sys\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.234150 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.234114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-lib-modules\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.234313 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.234280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-lib-modules\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.234313 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.234152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-podres\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.234573 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.234161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-sys\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.234573 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.234132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-proc\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.244357 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.244313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e-kube-api-access-ldcpm\") pod \"perf-node-gather-daemonset-97kh6\" (UID: \"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e\") " pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.394560 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.394232 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:51.629626 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.629588 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6"] Jul 20 23:39:51.641935 ip-10-0-134-37 kubenswrapper[2574]: W0720 23:39:51.641888 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f108e7a_7dd0_41c8_a3fa_6ee48bf4a69e.slice/crio-92a390926e51fc272c213526719f1c7c11007dffa0a7213b07d04c8c5744a107 WatchSource:0}: Error finding container 92a390926e51fc272c213526719f1c7c11007dffa0a7213b07d04c8c5744a107: Status 404 returned error can't find the container with id 92a390926e51fc272c213526719f1c7c11007dffa0a7213b07d04c8c5744a107 Jul 20 23:39:51.801162 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.801122 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/1.log" Jul 20 23:39:51.807157 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:51.807129 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-ncfsk_41a5f424-012a-41c7-ad86-424296e7b06d/console-operator/2.log" Jul 20 23:39:52.317591 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:52.317562 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5999c55d79-zdm9r_dce6f6e6-d4e2-4d07-a5c1-678372281016/console/0.log" Jul 20 23:39:52.351903 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:52.351871 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-d76hb_842f43cf-14e2-4ec6-b599-eedac3ed6b1a/download-server/0.log" Jul 20 23:39:52.646777 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:52.646746 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" event={"ID":"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e","Type":"ContainerStarted","Data":"55e6f3ef81647e3269530d100f69ae2d5b0e867b2f83e231abbec43aa4835d44"} Jul 20 23:39:52.646959 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:52.646796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" event={"ID":"7f108e7a-7dd0-41c8-a3fa-6ee48bf4a69e","Type":"ContainerStarted","Data":"92a390926e51fc272c213526719f1c7c11007dffa0a7213b07d04c8c5744a107"} Jul 20 23:39:52.646959 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:52.646827 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:52.667247 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:52.667174 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" podStartSLOduration=1.6671522730000001 podStartE2EDuration="1.667152273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 23:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 23:39:52.663323803 +0000 UTC m=+2782.159778240" watchObservedRunningTime="2026-07-20 23:39:52.667152273 +0000 UTC m=+2782.163606698" Jul 20 23:39:52.929437 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:52.929341 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-s84pf_3eed9716-0a22-48ec-aab4-850e2656a381/volume-data-source-validator/0.log" Jul 20 23:39:53.997385 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:53.997359 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lzh55_cba5b0a1-8585-46be-9dad-bc766849f281/dns/0.log" Jul 20 23:39:54.017318 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:54.017283 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lzh55_cba5b0a1-8585-46be-9dad-bc766849f281/kube-rbac-proxy/0.log" Jul 20 23:39:54.133767 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:54.133728 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjsrr_feb100de-3890-47c2-868d-79a783d51b47/dns-node-resolver/0.log" Jul 20 23:39:54.733208 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:54.733140 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vwqjp_6c0a5da1-61a9-4426-9646-800e40d3482c/node-ca/0.log" Jul 20 23:39:55.806631 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:55.806601 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-pnlcd_81a76f81-b28a-4956-abff-92d4a02a2a0c/kube-auth-proxy/0.log" Jul 20 23:39:56.636237 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:56.636210 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jlh2g_efb0d356-51ab-4dd1-b549-084d6c826bd8/serve-healthcheck-canary/0.log" Jul 20 23:39:57.165465 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:57.165436 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-flgnm_b03b1e37-25d1-4b71-a98b-ca2f6ec2ec88/insights-operator/0.log" Jul 20 23:39:57.324206 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:57.324171 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2dbp_5e442cd5-5bdd-4c94-a841-738395c275cc/kube-rbac-proxy/0.log" Jul 20 23:39:57.344001 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:57.343971 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2dbp_5e442cd5-5bdd-4c94-a841-738395c275cc/exporter/0.log" Jul 20 23:39:57.367344 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:57.367310 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x2dbp_5e442cd5-5bdd-4c94-a841-738395c275cc/extractor/0.log" Jul 20 23:39:58.666126 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:58.666091 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-95lz9/perf-node-gather-daemonset-97kh6" Jul 20 23:39:59.873165 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:59.873121 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-vt2g2_8cb9248c-af87-4f14-82f8-e2be59923b56/manager/0.log" Jul 20 23:39:59.900551 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:59.900520 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-ccxxt_f125e432-a510-404a-9039-e45602ab288b/manager/0.log" Jul 20 23:39:59.923413 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:39:59.923386 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-699fd79659-ccxxt_f125e432-a510-404a-9039-e45602ab288b/copy-manifests/0.log" Jul 20 23:40:01.536905 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:01.536830 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-87cc478df-p4pg2_ff749a45-0f9d-41a0-9b8e-2e2e85225f69/manager/0.log" Jul 20 23:40:02.073453 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:02.073424 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-pnlcd_81a76f81-b28a-4956-abff-92d4a02a2a0c/kube-auth-proxy/0.log" Jul 20 23:40:03.524100 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:03.524014 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85d6bc8fc7-pnlcd_81a76f81-b28a-4956-abff-92d4a02a2a0c/kube-auth-proxy/0.log" Jul 20 23:40:06.560981 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:06.560946 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-pdxhm_86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5/migrator/0.log" Jul 20 23:40:06.581940 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:06.581901 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-pdxhm_86d2e1d5-ec00-4b5a-9ed2-cefb3d4ac3b5/graceful-termination/0.log" Jul 20 23:40:06.949838 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:06.949804 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-84s25_077d3a13-d78c-4672-84c2-7dd2a551f322/kube-storage-version-migrator-operator/1.log" Jul 20 23:40:06.950834 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:06.950813 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-84s25_077d3a13-d78c-4672-84c2-7dd2a551f322/kube-storage-version-migrator-operator/0.log" Jul 20 23:40:08.003425 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:08.003396 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bsqhr_9c326231-6e02-4973-aa69-116110666151/kube-multus-additional-cni-plugins/0.log" Jul 20 23:40:08.024482 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:08.024451 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bsqhr_9c326231-6e02-4973-aa69-116110666151/egress-router-binary-copy/0.log" Jul 20 23:40:08.049679 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:08.049648 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bsqhr_9c326231-6e02-4973-aa69-116110666151/cni-plugins/0.log" Jul 20 23:40:08.074025 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:08.074000 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bsqhr_9c326231-6e02-4973-aa69-116110666151/bond-cni-plugin/0.log" Jul 20 23:40:08.097970 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:08.097936 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bsqhr_9c326231-6e02-4973-aa69-116110666151/routeoverride-cni/0.log" Jul 20 23:40:08.117751 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:08.117718 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bsqhr_9c326231-6e02-4973-aa69-116110666151/whereabouts-cni-bincopy/0.log" Jul 20 23:40:08.137464 ip-10-0-134-37 kubenswrapper[2574]: I0720 23:40:08.137432 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bsqhr_9c326231-6e02-4973-aa69-116110666151/whereabouts-cni/0.log"