May 20 13:48:01.922535 ip-10-0-141-251 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 13:48:01.922546 ip-10-0-141-251 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 13:48:01.922556 ip-10-0-141-251 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 13:48:01.922866 ip-10-0-141-251 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 13:48:11.958821 ip-10-0-141-251 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 20 13:48:11.958842 ip-10-0-141-251 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 64b6186b0e014449ad363ab4fba3fee7 --
May 20 13:50:15.245178 ip-10-0-141-251 systemd[1]: Starting Kubernetes Kubelet...
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.737706    2554 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741505    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741519    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741524    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741527    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741530    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741533    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741537    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:15.771480 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741540    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741543    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741546    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741549    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741551    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741554    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741557    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741560    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741562    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741565    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741569    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741572    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741575    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741577    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741579    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741582    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741585    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741587    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741590    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:15.772346 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741593    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741595    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741598    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741600    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741603    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741606    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741609    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741612    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741614    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741618    2554 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741621    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741625    2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741628    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741631    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741634    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741637    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741639    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741642    2554 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741644    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741647    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:15.773134 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741650    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741652    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741656    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741660    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741662    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741665    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741668    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741671    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741673    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741676    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741679    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741682    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741684    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741687    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741689    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741692    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741695    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741697    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741700    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:15.773694 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741704    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741708    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741710    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741713    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741715    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741718    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741720    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741723    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741727    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741729    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741732    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741735    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741737    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741739    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741743    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741747    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741750    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741753    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741755    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741758    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:15.774250 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.741761    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742179    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742185    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742188    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742191    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742194    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742197    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742200    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742203    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742206    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742209    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742212    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742214    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742229    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742232    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742235    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742238    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742241    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742243    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742246    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:15.774914 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742248    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742251    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742254    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742256    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742259    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742262    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742265    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742267    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742270    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742274    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742278    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742281    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742284    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742287    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742289    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742292    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742295    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742298    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742301    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742304    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:15.775499 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742306    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742309    2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742312    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742315    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742317    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742321    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742324    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742326    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742329    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742331    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742334    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742336    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742339    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742341    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742344    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742346    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742349    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742351    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742354    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:15.776303 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742357    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742359    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742362    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742364    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742366    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742370    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742373    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742375    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742378    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742381    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742384    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742386    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742389    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742391    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742395    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742398    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742400    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742403    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742406    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:15.776795 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742408    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742411    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742414    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742417    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742419    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742422    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742424    2554 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742427    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.742429    2554 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743075    2554 flags.go:64] FLAG: --address="0.0.0.0"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743085    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743092    2554 flags.go:64] FLAG: --anonymous-auth="true"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743097    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743101    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743105    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743109    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743114    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743117    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743120    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743124    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743127    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743130    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743133    2554 flags.go:64] FLAG: --cgroup-root=""
May 20 13:50:15.901989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743137    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743139    2554 flags.go:64] FLAG: --client-ca-file=""
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743146    2554 flags.go:64] FLAG: --cloud-config=""
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743149    2554 flags.go:64] FLAG: --cloud-provider="external"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743151    2554 flags.go:64] FLAG: --cluster-dns="[]"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743157    2554 flags.go:64] FLAG: --cluster-domain=""
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743160    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743163    2554 flags.go:64] FLAG: --config-dir=""
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743166    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743170    2554 flags.go:64] FLAG: --container-log-max-files="5"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743174    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743178    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743181    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743184    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743188    2554 flags.go:64] FLAG: --contention-profiling="false"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743191    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743194    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743205    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743209    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743213    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743216    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743219    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743222    2554 flags.go:64] FLAG: --enable-load-reader="false"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743226    2554 flags.go:64] FLAG: --enable-server="true"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743228    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 13:50:15.903396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743233    2554 flags.go:64] FLAG: --event-burst="100"
May 20 13:50:15.845398 ip-10-0-141-251 systemd[1]: Started Kubernetes Kubelet.
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743236    2554 flags.go:64] FLAG: --event-qps="50"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743239    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743243    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743246    2554 flags.go:64] FLAG: --eviction-hard=""
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743250    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743253    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743256    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743259    2554 flags.go:64] FLAG: --eviction-soft=""
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743263    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743267    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743270    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743273    2554 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743276    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743279    2554 flags.go:64] FLAG: --fail-swap-on="true"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743282    2554 flags.go:64] FLAG: --feature-gates=""
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743286    2554 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743289    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743292    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743295    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743299    2554 flags.go:64] FLAG: --healthz-port="10248"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743302    2554 flags.go:64] FLAG: --help="false"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743305    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-141-251.ec2.internal"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743308    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 13:50:15.996093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743311    2554 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743314    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743317    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743321    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743324    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743327    2554 flags.go:64] FLAG: --image-service-endpoint=""
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743330    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743332    2554 flags.go:64] FLAG: --kube-api-burst="100"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743335    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743339    2554 flags.go:64] FLAG: --kube-api-qps="50"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743342    2554 flags.go:64] FLAG: --kube-reserved=""
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743345    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743348    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743351    2554 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743355    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743358    2554 flags.go:64] FLAG: --lock-file=""
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743361    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743366    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743369    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743374    2554 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743377    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743380    2554 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743383    2554 flags.go:64] FLAG: --logging-format="text"
May 20 13:50:15.997489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743386    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743389    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743393    2554 flags.go:64] FLAG: --manifest-url=""
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743396    2554 flags.go:64] FLAG: --manifest-url-header=""
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743400    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743404    2554 flags.go:64] FLAG: --max-open-files="1000000"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743409    2554 flags.go:64] FLAG: --max-pods="110"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743412    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743415    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743418    2554 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743421    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743424    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743427    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743430    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743441    2554 flags.go:64] FLAG: --node-status-max-images="50"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743444    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743447    2554 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743450    2554 flags.go:64] FLAG: --pod-cidr=""
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743453    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743460    2554 flags.go:64] FLAG: --pod-manifest-path=""
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743463    2554 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743466    2554 flags.go:64] FLAG: --pods-per-core="0"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743469    2554 flags.go:64] FLAG: --port="10250"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743472    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 13:50:15.998198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743475    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02373e0dc62cc37e5"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743479    2554 flags.go:64] FLAG: --qos-reserved=""
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743483    2554 flags.go:64] FLAG: --read-only-port="10255"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743486    2554 flags.go:64] FLAG: --register-node="true"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743489    2554 flags.go:64] FLAG: --register-schedulable="true"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743492    2554 flags.go:64] FLAG: --register-with-taints=""
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743496    2554 flags.go:64] FLAG: --registry-burst="10"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743499    2554 flags.go:64] FLAG: --registry-qps="5"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743502    2554 flags.go:64] FLAG: --reserved-cpus=""
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743504    2554 flags.go:64] FLAG: --reserved-memory=""
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743508    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743512    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743515    2554 flags.go:64] FLAG: --rotate-certificates="false"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743518    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743521    2554 flags.go:64] FLAG: --runonce="false"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743524    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743528    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743531    2554 flags.go:64] FLAG: --seccomp-default="false"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743534    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743537    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743540    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743544    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743547    2554 flags.go:64] FLAG: --storage-driver-password="root"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743552    2554 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743555    2554 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743558    2554 flags.go:64] FLAG: --storage-driver-user="root"
May 20 13:50:15.999979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743561    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743564    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743567    2554 flags.go:64] FLAG: --system-cgroups=""
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743570    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743576    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743579    2554 flags.go:64] FLAG: --tls-cert-file=""
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743582    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743587    2554 flags.go:64] FLAG: --tls-min-version=""
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743591    2554 flags.go:64] FLAG: --tls-private-key-file=""
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743594    2554 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743597    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743600    2554 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743603    2554 flags.go:64] FLAG: --v="2"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743608    2554 flags.go:64] FLAG: --version="false"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743612    2554 flags.go:64] FLAG: --vmodule=""
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743624    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.743627    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743736    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743741    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743744    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743747    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743751    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743754    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:16.001018 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743758    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743760    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743763    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743767    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743771    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743776    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743781    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743784    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743786    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743789    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743792    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743795    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743798    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743800    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743803    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743806    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743809    2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743813    2554 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743815    2554 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:16.002063 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743818    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743820    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743823    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743826    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743828    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743831    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743833    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743850    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743857    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743860    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743862    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743865    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743868    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743871    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743874    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743877    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743879    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743882    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743885    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:16.002761 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743889    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743892    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743894    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743897    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743900    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743903    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743905    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743908    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743910    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743913    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743915    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743920    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743922    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743925    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743928    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743931    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743933    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743936    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743938    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743941    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:16.003801 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743943    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743946    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743949    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743951    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743954    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743956    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743959    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743963    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743965    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743968    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743971    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743973    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743977    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743980    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743982    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743985    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743988    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743990    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743993    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743995    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:16.005581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.743998    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.744000    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.744006    2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.751479    2554 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.751511    2554 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751598    2554 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751605    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751613    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751617    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751623    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751627    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751631    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751637    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751641    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751647    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751651    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:16.006131 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751658    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751662    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751668    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751671    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751675    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751679    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751683    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751691    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751696    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751704    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751708    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751715    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751719    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751725    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751729    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751735    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751739    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751742    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751748    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751754    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:16.006537 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751761    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751765    2554 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751772    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751776    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751781    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751785    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751789    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751795    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751799    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751804    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751810    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751818    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751822    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751830    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751867    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751874    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751878    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751882    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751885    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:16.007041 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751888    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751895    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751901    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751907    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751912    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751918    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751922    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751926    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751932    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751936    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751943    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751947    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751954    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751959    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751966    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751970    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751975    2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751979    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751985    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:16.007511 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751990    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.751994    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752000    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752004    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752010    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752014    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752020    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752024    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752028    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752034    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752038    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752043    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752047    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752051    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752057    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752062    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:16.008035 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752069    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.752078    2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752262    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752272    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752276    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752281    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752285    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752289    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752293    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752297    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752300    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752307    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752312    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752318    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752322    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:16.008432 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752328    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752332    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752335    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752341    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752345    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752351    2554 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752355    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752361    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752367    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752375    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752379    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752385    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752390    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752396    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752400    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752406    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752410    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752418    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752423    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752430    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:16.008800 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752434    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752440    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752444    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752448    2554 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752454    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752458    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752465    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752469    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752475    2554 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752479    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752973    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752981    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752985    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752992    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.752996    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753002    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753006    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753012    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753016    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753023    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:16.009302 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753027    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753030    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753036    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753040    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753043    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753049    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753053    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753058    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753063    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753071    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753075    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753081    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753085    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753091    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753095    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753099    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753103    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753105    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753114    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753118    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:16.009785 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753124    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753128    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753134    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753143    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753150    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753153    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753159    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753163    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753169    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753173    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753177    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753183    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:15.753187    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.753197    2554 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.754011    2554 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 13:50:16.010274 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.756516    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.757539    2554 server.go:1019] "Starting client certificate rotation"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.757641    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.758554    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.782861    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.791298    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.802480    2554 log.go:25] "Validated CRI v1 runtime API"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.808313    2554 log.go:25] "Validated CRI v1 image API"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.809437    2554 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.815661    2554 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b6ee1c3-0e93-42a8-8922-d8821f9909fb:/dev/nvme0n1p3 a250d2cf-f7d6-4960-a2a1-51188a292420:/dev/nvme0n1p4]
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.815680    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 13:50:16.010638 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.820089    2554 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.820647    2554 manager.go:217] Machine: {Timestamp:2026-05-20 13:50:15.820121674 +0000 UTC m=+0.442846733 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112181 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eea8e53f93e13d0c4fe7e3582499b SystemUUID:ec2eea8e-53f9-3e13-d0c4-fe7e3582499b BootID:64b6186b-0e01-4449-ad36-3ab4fba3fee7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a0:3b:6f:f8:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a0:3b:6f:f8:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:cd:a9:22:01:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.821528    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.821615    2554 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.823409    2554 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.823433    2554 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.823584    2554 topology_manager.go:138] "Creating topology manager with none policy"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.823592    2554 container_manager_linux.go:306] "Creating device plugin manager"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.823605    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.823620    2554 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.825776    2554 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.825912    2554 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.828359    2554 kubelet.go:491] "Attempting to sync node with API server"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.828373    2554 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.828387    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.828397    2554 kubelet.go:397] "Adding apiserver pod source"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.828406    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.829598    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:16.010967 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.829615    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.832774    2554 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.837655    2554 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838889    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838923    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838933    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838947    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838956    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838962    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838968    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838973    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838980    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838987    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.838996    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.839190    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.841296    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.841310    2554 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:15.841423    2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:15.841507    2554 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.844644    2554 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.844681    2554 server.go:1295] "Started kubelet"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.845106    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.845724    2554 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.845781    2554 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.847025    2554 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.847871    2554 server.go:317] "Adding debug handlers to kubelet server"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.852310    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.852607    2554 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.852753    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.852879    2554 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.852903    2554 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:15.852971    2554 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853021    2554 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853099    2554 reconstruct.go:97] "Volume reconstruction finished"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853107    2554 reconciler.go:26] "Reconciler: start to sync state"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853449    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853465    2554 factory.go:55] Registering systemd factory
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853475    2554 factory.go:223] Registration of the systemd container factory successfully
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:15.852554    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-251.ec2.internal.18b14a4b7b988cd2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-251.ec2.internal,UID:ip-10-0-141-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-251.ec2.internal,},FirstTimestamp:2026-05-20 13:50:15.84465429 +0000 UTC m=+0.467379348,LastTimestamp:2026-05-20 13:50:15.84465429 +0000 UTC m=+0.467379348,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-251.ec2.internal,}"
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853674    2554 factory.go:153] Registering CRI-O factory
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853687    2554 factory.go:223] Registration of the crio container factory successfully
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853707    2554 factory.go:103] Registering Raw factory
May 20 13:50:16.011451 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:15.853709    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.012509 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.853721    2554 manager.go:1196] Started watching for new ooms in manager
May 20 13:50:16.012509 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:15.854064    2554 manager.go:319] Starting recovery of all containers
May 20 13:50:16.012509 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:15.858602    2554 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 20 13:50:16.012509 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:15.859184    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 20 13:50:16.043360 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.042944    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2srv9"
May 20 13:50:16.043360 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.042970    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2srv9"
May 20 13:50:16.045304 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.045265    2554 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 13:50:16.045416 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.045337    2554 manager.go:324] Recovery completed
May 20 13:50:16.051316 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.051288    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.055425 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.055400    2554 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
May 20 13:50:16.058385 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.058371    2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:16.060625 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.060609    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:16.060685 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.060640    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:16.060685 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.060651    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:16.061172 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.061153    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 13:50:16.061172 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.061168    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 13:50:16.061285 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.061185    2554 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:16.061554 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.061536    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
May 20 13:50:16.063733 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.063721    2554 policy_none.go:49] "None policy: Start"
May 20 13:50:16.063781 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.063739    2554 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 13:50:16.063781 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.063751    2554 state_mem.go:35] "Initializing new in-memory state store"
May 20 13:50:16.105546 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.105522    2554 manager.go:341] "Starting Device Plugin manager"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.105566    2554 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.105579    2554 server.go:85] "Starting device plugin registration server"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.105884    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.105896    2554 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.105987    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.106098    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.106107    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.106627    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 13:50:16.116458 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.106661    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.171203 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.171120    2554 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 13:50:16.171203 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.171151    2554 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 13:50:16.171203 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.171171    2554 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 13:50:16.171203 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.171178    2554 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 13:50:16.171445 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.171218    2554 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 13:50:16.173608 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.173585    2554 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:16.206599 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.206568    2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:16.208485 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.208464    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:16.208609 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.208500    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:16.208609 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.208513    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:16.208609 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.208544    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-251.ec2.internal"
May 20 13:50:16.220219 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.220197    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-251.ec2.internal"
May 20 13:50:16.220219 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.220223    2554 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-251.ec2.internal\": node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.238978 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.238944    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.272022 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.271977    2554 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"]
May 20 13:50:16.272167 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.272070    2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:16.273052 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.273032    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:16.273157 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.273061    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:16.273157 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.273072    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:16.274347 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.274307    2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:16.274460 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.274444    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.274497 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.274473    2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:16.275058 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.275034    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:16.275058 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.275063    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:16.275192 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.275074    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:16.275192 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.275041    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:16.275192 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.275138    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:16.275192 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.275150    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:16.276233 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.276218    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.276322 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.276241    2554 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:16.276877 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.276863    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:16.276972 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.276887    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:16.276972 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.276900    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:16.300622 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.300591    2554 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-251.ec2.internal\" not found" node="ip-10-0-141-251.ec2.internal"
May 20 13:50:16.305063 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.305043    2554 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-251.ec2.internal\" not found" node="ip-10-0-141-251.ec2.internal"
May 20 13:50:16.339177 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.339147    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.439760 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.439688    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.443913 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.443894    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a05246c2f0322bd497a7601c61ef93cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"a05246c2f0322bd497a7601c61ef93cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.443962 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.443922    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a05246c2f0322bd497a7601c61ef93cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"a05246c2f0322bd497a7601c61ef93cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.443962 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.443939    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f62113004acf442450518e8b488b56e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"0f62113004acf442450518e8b488b56e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.540148 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.540118    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.544329 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.544307    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a05246c2f0322bd497a7601c61ef93cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"a05246c2f0322bd497a7601c61ef93cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.544380 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.544341    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a05246c2f0322bd497a7601c61ef93cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"a05246c2f0322bd497a7601c61ef93cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.544414 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.544379    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f62113004acf442450518e8b488b56e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"0f62113004acf442450518e8b488b56e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.544414 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.544397    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a05246c2f0322bd497a7601c61ef93cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"a05246c2f0322bd497a7601c61ef93cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.544474 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.544431    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f62113004acf442450518e8b488b56e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-251.ec2.internal\" (UID: \"0f62113004acf442450518e8b488b56e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.544474 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.544433    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a05246c2f0322bd497a7601c61ef93cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal\" (UID: \"a05246c2f0322bd497a7601c61ef93cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.604615 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.604583    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.607528 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.607498    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"
May 20 13:50:16.640817 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.640780    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.741344 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.741250    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.757741 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.757724    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 13:50:16.757918 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.757903    2554 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 13:50:16.842335 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.842295    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:16.852622 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.852597    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 13:50:16.869639 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.869609    2554 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:16.891317 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.891285    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bh68r"
May 20 13:50:16.899421 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:16.899394    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bh68r"
May 20 13:50:16.942675 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:16.942642    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:17.043186 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:17.043155    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:17.055628 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.055586    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 13:45:15 +0000 UTC" deadline="2027-12-19 22:32:59.693304531 +0000 UTC"
May 20 13:50:17.055628 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.055622    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13880h42m42.637686454s"
May 20 13:50:17.113696 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.113678    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:50:17.143724 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:17.143688    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:17.175134 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.175082    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"a05246c2f0322bd497a7601c61ef93cb","Type":"ContainerStarted","Data":"42e07f4906353fcef134166b05679de9783bb75df5ae32397d1343b53e1ff669"}
May 20 13:50:17.176046 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.176026    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" event={"ID":"0f62113004acf442450518e8b488b56e","Type":"ContainerStarted","Data":"3e811097d290d3c55362fd2def2955ef27a645d50b5e1d4b6d0f1e91257c5d63"}
May 20 13:50:17.244267 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:17.244229    2554 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-251.ec2.internal\" not found"
May 20 13:50:17.306393 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.306309    2554 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:17.316697 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.316669    2554 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:17.353080 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.353046    2554 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal"
May 20 13:50:17.367170 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.367144    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:17.368183 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.368167    2554 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal"
May 20 13:50:17.377024 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.377000    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:17.486936 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.486906    2554 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:17.829671 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.829636    2554 apiserver.go:52] "Watching apiserver"
May 20 13:50:17.841013 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.840974    2554 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 13:50:17.841410 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.841380    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7cqn4","openshift-multus/multus-n9nk9","openshift-multus/network-metrics-daemon-n4cgv","openshift-network-diagnostics/network-check-target-z9p8j","kube-system/konnectivity-agent-k2wv5","kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6","openshift-image-registry/node-ca-fg6nm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal","openshift-multus/multus-additional-cni-plugins-56jr7","openshift-network-operator/iptables-alerter-5tbmv","openshift-ovn-kubernetes/ovnkube-node-xd45j","openshift-cluster-node-tuning-operator/tuned-lh8dv"]
May 20 13:50:17.843383 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.843364    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.845039 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.845018    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.846040 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.846013    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 13:50:17.846040 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.846023    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rllbv\""
May 20 13:50:17.846181 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.846016    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 13:50:17.846181 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.846016    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 13:50:17.847154 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.847108    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 13:50:17.847489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.847467    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 13:50:17.847594 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.847562    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 13:50:17.848198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.848177    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:17.848322 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:17.848285    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:17.848979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.848588    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lckns\""
May 20 13:50:17.848979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.848635    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 13:50:17.850298 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.849943    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:17.850298 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.849966    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:17.850298 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:17.850121    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:17.852106 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.852085    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 13:50:17.852216 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.852194    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qh7ft\""
May 20 13:50:17.852373 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.852353    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 13:50:17.852963 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.852457    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:17.852963 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.852526    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:17.854120 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.854096    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.854852 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.854824    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 13:50:17.854974 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.854957    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 13:50:17.855417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.855065    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rltdz\""
May 20 13:50:17.855417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.855280    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 13:50:17.855417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.855329    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 13:50:17.855417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.855379    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 13:50:17.855647 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.855457    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:17.855647 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.855478    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8snb7\""
May 20 13:50:17.856287 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.856267    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 13:50:17.856387 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.856367    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kcbqn\""
May 20 13:50:17.856665 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.856619    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 13:50:17.857080 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.856994    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:17.857571 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.857549    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:17.857652 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.857596    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 13:50:17.857652 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.857645    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2xvnw\""
May 20 13:50:17.857941 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.857922    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 13:50:17.858863 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.858589    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:17.859177 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.859146    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 13:50:17.859266 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.859219    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 13:50:17.859494 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.859425    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hvvlr\""
May 20 13:50:17.859494 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.859432    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 13:50:17.859728 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.859711    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 13:50:17.859857 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.859821    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 13:50:17.860072 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.859967    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 13:50:17.860712 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.860693    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 13:50:17.860896 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.860876    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m5jvj\""
May 20 13:50:17.861171 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.861001    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:17.890306 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.890281    2554 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:17.902156 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.902124    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:16 +0000 UTC" deadline="2028-02-17 01:38:44.199227826 +0000 UTC"
May 20 13:50:17.902156 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.902154    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15299h48m26.297076944s"
May 20 13:50:17.951428 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-cni-multus\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951428 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951433    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-daemon-config\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951619 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951501    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-etc-kubernetes\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951619 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951537    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkqn\" (UniqueName: \"kubernetes.io/projected/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-kube-api-access-xbkqn\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951619 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951555    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-host\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:17.951619 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951578    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm5k\" (UniqueName: \"kubernetes.io/projected/b8c29338-36d1-4d0d-942e-6b5312426efc-kube-api-access-qfm5k\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:17.951795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-system-cni-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951638    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-cni-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951654    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:17.951795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951670    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgdgf\" (UniqueName: \"kubernetes.io/projected/d10bae7d-9cef-4c47-9918-5c9748ff88a2-kube-api-access-lgdgf\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.951795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951691    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-cni-binary-copy\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951712    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-cni-bin\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.951795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951741    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-multus-certs\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951832    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951889    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951928    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-sys-fs\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951955    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-socket-dir-parent\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.951978    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-conf-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952003    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f817fca4-539b-4c30-8739-dd4417db0223-hosts-file\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952027    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-k8s-cni-cncf-io\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952051    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-netns\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952074    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-kubelet\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-hostroot\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952140    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b32bac4-4900-4fb5-bfe7-191ce7aab365-konnectivity-ca\") pod \"konnectivity-agent-k2wv5\" (UID: \"1b32bac4-4900-4fb5-bfe7-191ce7aab365\") " pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:17.952166 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952161    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtq9g\" (UniqueName: \"kubernetes.io/projected/f817fca4-539b-4c30-8739-dd4417db0223-kube-api-access-gtq9g\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952183    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-system-cni-dir\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952205    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-os-release\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952220    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-socket-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952242    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-serviceca\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952266    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952287    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-device-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952305    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b32bac4-4900-4fb5-bfe7-191ce7aab365-agent-certs\") pod \"konnectivity-agent-k2wv5\" (UID: \"1b32bac4-4900-4fb5-bfe7-191ce7aab365\") " pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f817fca4-539b-4c30-8739-dd4417db0223-tmp-dir\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952342    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952378    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952411    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-os-release\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952429    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqsw5\" (UniqueName: \"kubernetes.io/projected/7e4366cf-e68e-4172-9240-ccdd09127623-kube-api-access-vqsw5\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952451    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-cnibin\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952491    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64pfd\" (UniqueName: \"kubernetes.io/projected/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-kube-api-access-64pfd\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952530    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cnibin\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:17.952653 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952557    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-registration-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.953199 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.952582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:17.953715 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:17.953695    2554 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 13:50:18.053040 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.052957    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-system-cni-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053040 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.052993    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-cni-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053040 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053010    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:18.053276 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.053120    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:18.053276 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053133    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-cni-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053276 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053164    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-cni-netd\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.053276 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053181    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-system-cni-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053276 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.053219    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:18.553175516 +0000 UTC m=+3.175900577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:18.053276 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053256    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-ovnkube-config\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.053276 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053280    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-cni-binary-copy\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053295    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-cni-bin\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053315    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-run\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053336    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-cni-bin\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053339    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-run-netns\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053384    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-socket-dir-parent\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053409    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-conf-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053437    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f817fca4-539b-4c30-8739-dd4417db0223-hosts-file\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053458    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-socket-dir-parent\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053463    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053512    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f817fca4-539b-4c30-8739-dd4417db0223-hosts-file\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053515    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-conf-dir\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053522    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-sys\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053576    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62665\" (UniqueName: \"kubernetes.io/projected/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-kube-api-access-62665\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.053602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053604    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-systemd\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053633    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-hostroot\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053634    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053671    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-hostroot\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053697    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6h8z\" (UniqueName: \"kubernetes.io/projected/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-kube-api-access-j6h8z\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053744    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-lib-modules\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053767    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-env-overrides\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053829    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-serviceca\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053878    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-modprobe-d\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysctl-d\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053931    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-etc-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053951    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-cni-binary-copy\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.053957    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-run-ovn-kubernetes\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054008    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f817fca4-539b-4c30-8739-dd4417db0223-tmp-dir\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054037    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-slash\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054261 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054256    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-cnibin\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054307    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f817fca4-539b-4c30-8739-dd4417db0223-tmp-dir\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054116    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-cnibin\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054568    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64pfd\" (UniqueName: \"kubernetes.io/projected/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-kube-api-access-64pfd\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054593    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysconfig\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054618    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-log-socket\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054621    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-serviceca\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054639    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-cni-bin\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054664    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054686    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32db493c-1a8f-4925-ada5-8f0562c64769-ovn-node-metrics-cert\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054708    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-ovnkube-script-lib\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054736    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-registration-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.054785 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054772    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-cni-multus\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054823    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkqn\" (UniqueName: \"kubernetes.io/projected/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-kube-api-access-xbkqn\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054862    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-registration-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054871    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-host\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054897    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm5k\" (UniqueName: \"kubernetes.io/projected/b8c29338-36d1-4d0d-942e-6b5312426efc-kube-api-access-qfm5k\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054917    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-host\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054924    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-iptables-alerter-script\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054948    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcscq\" (UniqueName: \"kubernetes.io/projected/32db493c-1a8f-4925-ada5-8f0562c64769-kube-api-access-tcscq\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054962    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-cni-multus\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.054983    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-systemd\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055008    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-host\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055033    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-tuned\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055055    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-systemd-units\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.055307 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055094    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-node-log\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055413    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055463    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-var-lib-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055490    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055531    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-multus-certs\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055566    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055615    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055674    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-sys-fs\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055698    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-sys-fs\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055723    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-system-cni-dir\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055761    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.055791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055773    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-multus-certs\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055798    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-tmp\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055823    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055831    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-kubelet\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055884    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-k8s-cni-cncf-io\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055910    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-netns\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055941    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-kubelet\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055970    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cni-binary-copy\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.055974    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b32bac4-4900-4fb5-bfe7-191ce7aab365-konnectivity-ca\") pod \"konnectivity-agent-k2wv5\" (UID: \"1b32bac4-4900-4fb5-bfe7-191ce7aab365\") " pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056007    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gtq9g\" (UniqueName: \"kubernetes.io/projected/f817fca4-539b-4c30-8739-dd4417db0223-kube-api-access-gtq9g\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056023    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-system-cni-dir\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056036    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-os-release\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056064    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-k8s-cni-cncf-io\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056064    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056099    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-kubernetes\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.056150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056130    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-socket-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056161    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-host-slash\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056191    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysctl-conf\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056218    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-var-lib-kubelet\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056251    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-device-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056282    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b32bac4-4900-4fb5-bfe7-191ce7aab365-agent-certs\") pod \"konnectivity-agent-k2wv5\" (UID: \"1b32bac4-4900-4fb5-bfe7-191ce7aab365\") " pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056314    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-ovn\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056317    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056347    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-os-release\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056379    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vqsw5\" (UniqueName: \"kubernetes.io/projected/7e4366cf-e68e-4172-9240-ccdd09127623-kube-api-access-vqsw5\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056410    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cnibin\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056434    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b32bac4-4900-4fb5-bfe7-191ce7aab365-konnectivity-ca\") pod \"konnectivity-agent-k2wv5\" (UID: \"1b32bac4-4900-4fb5-bfe7-191ce7aab365\") " pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056487    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-run-netns\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056519    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-host-var-lib-kubelet\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056436    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.056556 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056551    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-daemon-config\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.056978 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056626    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-os-release\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.056978 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056829    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-socket-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.056978 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056864    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-device-dir\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.056978 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.056904    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-etc-kubernetes\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057170    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lgdgf\" (UniqueName: \"kubernetes.io/projected/d10bae7d-9cef-4c47-9918-5c9748ff88a2-kube-api-access-lgdgf\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057237    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-multus-daemon-config\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057320    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-etc-kubernetes\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057323    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e4366cf-e68e-4172-9240-ccdd09127623-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057498    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d10bae7d-9cef-4c47-9918-5c9748ff88a2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057632    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057724    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-os-release\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.058895 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.057773    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d10bae7d-9cef-4c47-9918-5c9748ff88a2-cnibin\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.061448 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.061424    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b32bac4-4900-4fb5-bfe7-191ce7aab365-agent-certs\") pod \"konnectivity-agent-k2wv5\" (UID: \"1b32bac4-4900-4fb5-bfe7-191ce7aab365\") " pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:18.062499 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.062236    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:18.062499 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.062259    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:18.062499 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.062273    2554 projected.go:194] Error preparing data for projected volume kube-api-access-tsr8m for pod openshift-network-diagnostics/network-check-target-z9p8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:18.062499 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.062339    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m podName:77da8e32-a72e-4ec4-96dc-97db3ede4765 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:18.562321104 +0000 UTC m=+3.185046167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tsr8m" (UniqueName: "kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m") pod "network-check-target-z9p8j" (UID: "77da8e32-a72e-4ec4-96dc-97db3ede4765") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:18.064214 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.064190    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64pfd\" (UniqueName: \"kubernetes.io/projected/2790ca0e-0d38-4d36-94bd-f9d07e2aba29-kube-api-access-64pfd\") pod \"node-ca-fg6nm\" (UID: \"2790ca0e-0d38-4d36-94bd-f9d07e2aba29\") " pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:18.064328 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.064227    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtq9g\" (UniqueName: \"kubernetes.io/projected/f817fca4-539b-4c30-8739-dd4417db0223-kube-api-access-gtq9g\") pod \"node-resolver-7cqn4\" (UID: \"f817fca4-539b-4c30-8739-dd4417db0223\") " pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:18.064811 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.064703    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm5k\" (UniqueName: \"kubernetes.io/projected/b8c29338-36d1-4d0d-942e-6b5312426efc-kube-api-access-qfm5k\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:18.064904 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.064857    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkqn\" (UniqueName: \"kubernetes.io/projected/575a7241-3e3e-4fbd-a215-e69cb73bf5a4-kube-api-access-xbkqn\") pod \"multus-n9nk9\" (UID: \"575a7241-3e3e-4fbd-a215-e69cb73bf5a4\") " pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.065828 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.065811    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqsw5\" (UniqueName: \"kubernetes.io/projected/7e4366cf-e68e-4172-9240-ccdd09127623-kube-api-access-vqsw5\") pod \"aws-ebs-csi-driver-node-v2dn6\" (UID: \"7e4366cf-e68e-4172-9240-ccdd09127623\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.066124 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.066101    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgdgf\" (UniqueName: \"kubernetes.io/projected/d10bae7d-9cef-4c47-9918-5c9748ff88a2-kube-api-access-lgdgf\") pod \"multus-additional-cni-plugins-56jr7\" (UID: \"d10bae7d-9cef-4c47-9918-5c9748ff88a2\") " pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.157957 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.157914    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6"
May 20 13:50:18.158141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.157978    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-etc-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158015    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-run-ovn-kubernetes\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158041    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-slash\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158052    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-etc-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158068    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysconfig\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.158141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158100    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-slash\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158123    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-run-ovn-kubernetes\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158131    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysconfig\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158143    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-log-socket\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158179    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-cni-bin\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158204    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158213    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-cni-bin\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158182    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-log-socket\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158221    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32db493c-1a8f-4925-ada5-8f0562c64769-ovn-node-metrics-cert\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158262    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158265    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-ovnkube-script-lib\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158298    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-iptables-alerter-script\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158315    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tcscq\" (UniqueName: \"kubernetes.io/projected/32db493c-1a8f-4925-ada5-8f0562c64769-kube-api-access-tcscq\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158342    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-systemd\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158367    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-host\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158392    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-tuned\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158415    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-systemd-units\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158436    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-node-log\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.158489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158460    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-var-lib-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158483    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158526    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-tmp\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158562    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-kubelet\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158594    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-kubernetes\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158628    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-host-slash\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158670    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysctl-conf\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158695    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-var-lib-kubelet\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158731    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-ovn\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158736    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-systemd\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158787    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-cni-netd\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158799    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-host\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158813    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-ovnkube-config\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158830    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-ovnkube-script-lib\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158859    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-run\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158866    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-iptables-alerter-script\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158911    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-systemd-units\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158917    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-run\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.159207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158960    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-kubernetes\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158974    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-var-lib-kubelet\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158694    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-node-log\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158989    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-run-netns\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.158992    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysctl-conf\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159036    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-cni-netd\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159044    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-kubelet\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159070    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-ovn\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159086    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-host-slash\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159103    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-host-run-netns\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159113    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-sys\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159138    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-62665\" (UniqueName: \"kubernetes.io/projected/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-kube-api-access-62665\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159163    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-systemd\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159189    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6h8z\" (UniqueName: \"kubernetes.io/projected/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-kube-api-access-j6h8z\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159214    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-lib-modules\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159219    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-var-lib-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159238    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-env-overrides\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159264    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-modprobe-d\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.160367 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159268    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-systemd\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159299    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysctl-d\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159379    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-modprobe-d\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159408    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-ovnkube-config\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159428    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-sysctl-d\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159475    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-sys\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159493    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/32db493c-1a8f-4925-ada5-8f0562c64769-run-openvswitch\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159570    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-lib-modules\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.161280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.159959    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/32db493c-1a8f-4925-ada5-8f0562c64769-env-overrides\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.161646 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.161327    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-etc-tuned\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.161646 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.161631    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-tmp\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.161744 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.161697    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/32db493c-1a8f-4925-ada5-8f0562c64769-ovn-node-metrics-cert\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.163758 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.163735    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9nk9"
May 20 13:50:18.166665 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.166646    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcscq\" (UniqueName: \"kubernetes.io/projected/32db493c-1a8f-4925-ada5-8f0562c64769-kube-api-access-tcscq\") pod \"ovnkube-node-xd45j\" (UID: \"32db493c-1a8f-4925-ada5-8f0562c64769\") " pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.167130 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.167110    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6h8z\" (UniqueName: \"kubernetes.io/projected/b78bf5cb-cabe-4493-9b8c-c3b0be0f493e-kube-api-access-j6h8z\") pod \"iptables-alerter-5tbmv\" (UID: \"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e\") " pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.167204 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.167163    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62665\" (UniqueName: \"kubernetes.io/projected/f2b4c2c0-f318-4ceb-838e-a44c73eb275a-kube-api-access-62665\") pod \"tuned-lh8dv\" (UID: \"f2b4c2c0-f318-4ceb-838e-a44c73eb275a\") " pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.173112 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.173092    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:18.178364 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.178347    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7cqn4"
May 20 13:50:18.185899 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.185877    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fg6nm"
May 20 13:50:18.193488 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.193456    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-56jr7"
May 20 13:50:18.200088 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.200064    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5tbmv"
May 20 13:50:18.207666 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.207635    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:18.211355 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.211334    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lh8dv"
May 20 13:50:18.562139 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.562091    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:18.562306 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.562216    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:18.562306 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.562281    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:19.562264124 +0000 UTC m=+4.184989169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:18.620607 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.620567    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32db493c_1a8f_4925_ada5_8f0562c64769.slice/crio-0066df61615915f04f7dce859d9c77a6381a287b1534d915c239a1859d21264b WatchSource:0}: Error finding container 0066df61615915f04f7dce859d9c77a6381a287b1534d915c239a1859d21264b: Status 404 returned error can't find the container with id 0066df61615915f04f7dce859d9c77a6381a287b1534d915c239a1859d21264b
May 20 13:50:18.621963 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.621881    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b4c2c0_f318_4ceb_838e_a44c73eb275a.slice/crio-f33fa22b0eb2a66457841fb6c78ed2a49353d0d02e2acb01a03927def8c5e816 WatchSource:0}: Error finding container f33fa22b0eb2a66457841fb6c78ed2a49353d0d02e2acb01a03927def8c5e816: Status 404 returned error can't find the container with id f33fa22b0eb2a66457841fb6c78ed2a49353d0d02e2acb01a03927def8c5e816
May 20 13:50:18.622626 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.622604    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf817fca4_539b_4c30_8739_dd4417db0223.slice/crio-53be104bcf40389fd2a6f0fc192d98017e6c7c274d90304e469d6363f02b3635 WatchSource:0}: Error finding container 53be104bcf40389fd2a6f0fc192d98017e6c7c274d90304e469d6363f02b3635: Status 404 returned error can't find the container with id 53be104bcf40389fd2a6f0fc192d98017e6c7c274d90304e469d6363f02b3635
May 20 13:50:18.623452 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.623428    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2790ca0e_0d38_4d36_94bd_f9d07e2aba29.slice/crio-99baf38062f5fa9ba6a3692b53b270c7f3f033af47c0b93a1761faa24c0d2cfc WatchSource:0}: Error finding container 99baf38062f5fa9ba6a3692b53b270c7f3f033af47c0b93a1761faa24c0d2cfc: Status 404 returned error can't find the container with id 99baf38062f5fa9ba6a3692b53b270c7f3f033af47c0b93a1761faa24c0d2cfc
May 20 13:50:18.624438 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.624381    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10bae7d_9cef_4c47_9918_5c9748ff88a2.slice/crio-fbe7392d7b97087324ebd9589582d97a91c2165facb139179af974969440f1bd WatchSource:0}: Error finding container fbe7392d7b97087324ebd9589582d97a91c2165facb139179af974969440f1bd: Status 404 returned error can't find the container with id fbe7392d7b97087324ebd9589582d97a91c2165facb139179af974969440f1bd
May 20 13:50:18.625920 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.625894    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575a7241_3e3e_4fbd_a215_e69cb73bf5a4.slice/crio-f46df4993bdf3a9d7520c5551d38349735842be752955be48975b56161cf2b94 WatchSource:0}: Error finding container f46df4993bdf3a9d7520c5551d38349735842be752955be48975b56161cf2b94: Status 404 returned error can't find the container with id f46df4993bdf3a9d7520c5551d38349735842be752955be48975b56161cf2b94
May 20 13:50:18.627765 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.627741    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78bf5cb_cabe_4493_9b8c_c3b0be0f493e.slice/crio-3fc77cfc9a97a63eb28fa00af45306ce796e0d7b7470a5405d66dd019cae6c97 WatchSource:0}: Error finding container 3fc77cfc9a97a63eb28fa00af45306ce796e0d7b7470a5405d66dd019cae6c97: Status 404 returned error can't find the container with id 3fc77cfc9a97a63eb28fa00af45306ce796e0d7b7470a5405d66dd019cae6c97
May 20 13:50:18.629488 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.629361    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b32bac4_4900_4fb5_bfe7_191ce7aab365.slice/crio-a54386bc4784a1eef43d28d601b6795a12d7f87259f12511c41c6fcd892bebf7 WatchSource:0}: Error finding container a54386bc4784a1eef43d28d601b6795a12d7f87259f12511c41c6fcd892bebf7: Status 404 returned error can't find the container with id a54386bc4784a1eef43d28d601b6795a12d7f87259f12511c41c6fcd892bebf7
May 20 13:50:18.631571 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:18.631548    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4366cf_e68e_4172_9240_ccdd09127623.slice/crio-a03686aea6d4f00560096a3845d0fe15c6e4c780f581e97ec7299242e1fe7d5b WatchSource:0}: Error finding container a03686aea6d4f00560096a3845d0fe15c6e4c780f581e97ec7299242e1fe7d5b: Status 404 returned error can't find the container with id a03686aea6d4f00560096a3845d0fe15c6e4c780f581e97ec7299242e1fe7d5b
May 20 13:50:18.663107 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.663076    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:18.663235 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.663210    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:18.663235 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.663230    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:18.663332 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.663244    2554 projected.go:194] Error preparing data for projected volume kube-api-access-tsr8m for pod openshift-network-diagnostics/network-check-target-z9p8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:18.663332 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:18.663308    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m podName:77da8e32-a72e-4ec4-96dc-97db3ede4765 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:19.663288862 +0000 UTC m=+4.286013911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsr8m" (UniqueName: "kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m") pod "network-check-target-z9p8j" (UID: "77da8e32-a72e-4ec4-96dc-97db3ede4765") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:18.902527 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.902496    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:16 +0000 UTC" deadline="2028-01-31 03:04:27.377771486 +0000 UTC"
May 20 13:50:18.902527 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:18.902524    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14893h14m8.475249725s"
May 20 13:50:19.193288 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.192514    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" event={"ID":"0f62113004acf442450518e8b488b56e","Type":"ContainerStarted","Data":"1d9828f40dd4f59b4f3f8079c37809528364d6765d902ab45610658bb1da666a"}
May 20 13:50:19.200882 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.198859    2554 generic.go:358] "Generic (PLEG): container finished" podID="a05246c2f0322bd497a7601c61ef93cb" containerID="9bdd6a6d68b34dc0f726c592ade324671096e8f477eaf454c54b2dc24485e2b0" exitCode=0
May 20 13:50:19.200882 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.199127    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"a05246c2f0322bd497a7601c61ef93cb","Type":"ContainerDied","Data":"9bdd6a6d68b34dc0f726c592ade324671096e8f477eaf454c54b2dc24485e2b0"}
May 20 13:50:19.206215 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.206159    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6" event={"ID":"7e4366cf-e68e-4172-9240-ccdd09127623","Type":"ContainerStarted","Data":"a03686aea6d4f00560096a3845d0fe15c6e4c780f581e97ec7299242e1fe7d5b"}
May 20 13:50:19.214929 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.214883    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5tbmv" event={"ID":"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e","Type":"ContainerStarted","Data":"3fc77cfc9a97a63eb28fa00af45306ce796e0d7b7470a5405d66dd019cae6c97"}
May 20 13:50:19.215269 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.215228    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-251.ec2.internal" podStartSLOduration=2.21521535 podStartE2EDuration="2.21521535s" podCreationTimestamp="2026-05-20 13:50:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:19.214542719 +0000 UTC m=+3.837267812" watchObservedRunningTime="2026-05-20 13:50:19.21521535 +0000 UTC m=+3.837940420"
May 20 13:50:19.233731 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.233695    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9nk9" event={"ID":"575a7241-3e3e-4fbd-a215-e69cb73bf5a4","Type":"ContainerStarted","Data":"f46df4993bdf3a9d7520c5551d38349735842be752955be48975b56161cf2b94"}
May 20 13:50:19.245959 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.245925    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fg6nm" event={"ID":"2790ca0e-0d38-4d36-94bd-f9d07e2aba29","Type":"ContainerStarted","Data":"99baf38062f5fa9ba6a3692b53b270c7f3f033af47c0b93a1761faa24c0d2cfc"}
May 20 13:50:19.256422 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.256341    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2wv5" event={"ID":"1b32bac4-4900-4fb5-bfe7-191ce7aab365","Type":"ContainerStarted","Data":"a54386bc4784a1eef43d28d601b6795a12d7f87259f12511c41c6fcd892bebf7"}
May 20 13:50:19.263830 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.263764    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7cqn4" event={"ID":"f817fca4-539b-4c30-8739-dd4417db0223","Type":"ContainerStarted","Data":"53be104bcf40389fd2a6f0fc192d98017e6c7c274d90304e469d6363f02b3635"}
May 20 13:50:19.265560 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.265514    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerStarted","Data":"fbe7392d7b97087324ebd9589582d97a91c2165facb139179af974969440f1bd"}
May 20 13:50:19.276928 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.276895    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lh8dv" event={"ID":"f2b4c2c0-f318-4ceb-838e-a44c73eb275a","Type":"ContainerStarted","Data":"f33fa22b0eb2a66457841fb6c78ed2a49353d0d02e2acb01a03927def8c5e816"}
May 20 13:50:19.294975 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.294905    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"0066df61615915f04f7dce859d9c77a6381a287b1534d915c239a1859d21264b"}
May 20 13:50:19.571340 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.571246    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:19.571496 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:19.571403    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:19.571496 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:19.571467    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:21.571449027 +0000 UTC m=+6.194174079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:19.671629 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:19.671586    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:19.671815 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:19.671777    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:19.671815 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:19.671796    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:19.671815 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:19.671807    2554 projected.go:194] Error preparing data for projected volume kube-api-access-tsr8m for pod openshift-network-diagnostics/network-check-target-z9p8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:19.672005 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:19.671884    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m podName:77da8e32-a72e-4ec4-96dc-97db3ede4765 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:21.671865688 +0000 UTC m=+6.294590734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsr8m" (UniqueName: "kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m") pod "network-check-target-z9p8j" (UID: "77da8e32-a72e-4ec4-96dc-97db3ede4765") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:20.172239 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:20.172205    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:20.172716 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:20.172334    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:20.172811 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:20.172785    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:20.172939 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:20.172918    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:20.323234 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:20.323199    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" event={"ID":"a05246c2f0322bd497a7601c61ef93cb","Type":"ContainerStarted","Data":"134469834c9bcf7ebd986702fddc20451e5ab5e9aa72dcc370e3a54ffef9f485"}
May 20 13:50:20.337484 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:20.337426    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-251.ec2.internal" podStartSLOduration=3.337406953 podStartE2EDuration="3.337406953s" podCreationTimestamp="2026-05-20 13:50:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:20.336254486 +0000 UTC m=+4.958979574" watchObservedRunningTime="2026-05-20 13:50:20.337406953 +0000 UTC m=+4.960132023"
May 20 13:50:21.588402 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:21.588360    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:21.588886 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:21.588545    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:21.588886 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:21.588614    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:25.588590025 +0000 UTC m=+10.211315227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:21.689340 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:21.689299    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:21.689538 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:21.689452    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:21.689538 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:21.689473    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:21.689538 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:21.689487    2554 projected.go:194] Error preparing data for projected volume kube-api-access-tsr8m for pod openshift-network-diagnostics/network-check-target-z9p8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:21.689695 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:21.689551    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m podName:77da8e32-a72e-4ec4-96dc-97db3ede4765 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:25.689533223 +0000 UTC m=+10.312258273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsr8m" (UniqueName: "kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m") pod "network-check-target-z9p8j" (UID: "77da8e32-a72e-4ec4-96dc-97db3ede4765") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:22.171505 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.171473    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:22.171699 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:22.171615    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:22.173436 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.173408    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:22.173601 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:22.173560    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:22.645691 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.645591    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-smc4h"]
May 20 13:50:22.654284 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.654255    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.654461 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:22.654339    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:22.697622 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.697561    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.697622 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.697619    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d5a03ba-819e-4e8c-b589-37e651c8f956-dbus\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.697899 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.697650    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d5a03ba-819e-4e8c-b589-37e651c8f956-kubelet-config\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.798898 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.798860    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.799086 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.798915    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d5a03ba-819e-4e8c-b589-37e651c8f956-dbus\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.799086 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.798945    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d5a03ba-819e-4e8c-b589-37e651c8f956-kubelet-config\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.799086 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.799048    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d5a03ba-819e-4e8c-b589-37e651c8f956-kubelet-config\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:22.799086 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:22.799046    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:22.799268 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:22.799129    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret podName:5d5a03ba-819e-4e8c-b589-37e651c8f956 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:23.299109683 +0000 UTC m=+7.921834728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret") pod "global-pull-secret-syncer-smc4h" (UID: "5d5a03ba-819e-4e8c-b589-37e651c8f956") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:22.799268 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:22.799157    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d5a03ba-819e-4e8c-b589-37e651c8f956-dbus\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:23.302794 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:23.302743    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:23.302996 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:23.302925    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:23.303072 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:23.303005    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret podName:5d5a03ba-819e-4e8c-b589-37e651c8f956 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:24.302984106 +0000 UTC m=+8.925709168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret") pod "global-pull-secret-syncer-smc4h" (UID: "5d5a03ba-819e-4e8c-b589-37e651c8f956") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:24.171464 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:24.171427    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:24.171952 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:24.171427    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:24.171952 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:24.171560    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:24.171952 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:24.171613    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:24.171952 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:24.171700    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:24.171952 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:24.171799    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:24.311628 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:24.311587    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:24.311802 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:24.311735    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:24.311877 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:24.311812    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret podName:5d5a03ba-819e-4e8c-b589-37e651c8f956 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:26.311798272 +0000 UTC m=+10.934523319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret") pod "global-pull-secret-syncer-smc4h" (UID: "5d5a03ba-819e-4e8c-b589-37e651c8f956") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:25.624515 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:25.624474    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:25.625016 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:25.624628    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:25.625016 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:25.624704    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:33.624684679 +0000 UTC m=+18.247409729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:25.725753 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:25.725712    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:25.726086 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:25.725960    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:25.726086 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:25.725986    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:25.726086 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:25.725999    2554 projected.go:194] Error preparing data for projected volume kube-api-access-tsr8m for pod openshift-network-diagnostics/network-check-target-z9p8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:25.726086 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:25.726066    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m podName:77da8e32-a72e-4ec4-96dc-97db3ede4765 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:33.726043962 +0000 UTC m=+18.348769033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsr8m" (UniqueName: "kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m") pod "network-check-target-z9p8j" (UID: "77da8e32-a72e-4ec4-96dc-97db3ede4765") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:26.173226 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:26.173184    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:26.173397 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:26.173283    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:26.173397 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:26.173323    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:26.173397 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:26.173321    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:26.173552 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:26.173405    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:26.173552 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:26.173499    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:26.330222 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:26.330180    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:26.330400 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:26.330356    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:26.330465 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:26.330437    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret podName:5d5a03ba-819e-4e8c-b589-37e651c8f956 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:30.330416379 +0000 UTC m=+14.953141432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret") pod "global-pull-secret-syncer-smc4h" (UID: "5d5a03ba-819e-4e8c-b589-37e651c8f956") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:28.171732 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:28.171696    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:28.171732 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:28.171716    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:28.172244 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:28.171696    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:28.172244 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:28.171824    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:28.172244 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:28.171925    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:28.172244 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:28.172019    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:30.172002 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:30.171968    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:30.172407 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:30.172004    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:30.172407 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:30.171968    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:30.172407 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:30.172078    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:30.172407 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:30.172133    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:30.172407 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:30.172218    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:30.360700 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:30.360665    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:30.360878 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:30.360824    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:30.360979 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:30.360897    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret podName:5d5a03ba-819e-4e8c-b589-37e651c8f956 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:38.360882449 +0000 UTC m=+22.983607503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret") pod "global-pull-secret-syncer-smc4h" (UID: "5d5a03ba-819e-4e8c-b589-37e651c8f956") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:32.172146 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:32.172108    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:32.172567 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:32.172111    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:32.172567 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:32.172236    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:32.172567 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:32.172109    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:32.172567 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:32.172341    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:32.172567 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:32.172452    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:33.684659 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:33.684613    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:33.685142 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:33.684769    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:33.685142 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:33.684857    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:49.684828333 +0000 UTC m=+34.307553379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:33.785330 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:33.785274    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:33.785529 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:33.785422    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:33.785529 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:33.785440    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:33.785529 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:33.785452    2554 projected.go:194] Error preparing data for projected volume kube-api-access-tsr8m for pod openshift-network-diagnostics/network-check-target-z9p8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:33.785529 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:33.785514    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m podName:77da8e32-a72e-4ec4-96dc-97db3ede4765 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:49.785495529 +0000 UTC m=+34.408220579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsr8m" (UniqueName: "kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m") pod "network-check-target-z9p8j" (UID: "77da8e32-a72e-4ec4-96dc-97db3ede4765") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:34.172075 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:34.172037    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:34.172262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:34.172037    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:34.172262 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:34.172158    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:34.172262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:34.172039    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:34.172397 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:34.172222    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:34.172397 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:34.172341    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:36.172249 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:36.172213    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:36.172700 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:36.172322    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:36.172700 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:36.172386    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:36.172700 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:36.172459    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:36.172700 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:36.172492    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:36.172700 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:36.172598    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:38.171978 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.171752    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:38.172859 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:38.172070    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:38.172859 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.171751    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:38.172859 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:38.172180    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:38.172859 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.171750    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:38.172859 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:38.172258    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:38.376639 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.376608    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2wv5" event={"ID":"1b32bac4-4900-4fb5-bfe7-191ce7aab365","Type":"ContainerStarted","Data":"25e5a9aa52e9745166d5e18fbc02c154a8f0b6e1fd98f9767b82aa7c9df2b954"}
May 20 13:50:38.378690 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.378656    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7cqn4" event={"ID":"f817fca4-539b-4c30-8739-dd4417db0223","Type":"ContainerStarted","Data":"b397ceb170e9d654eea410bc8832359f1019cdfd50bb9f97d29e9d9807d333da"}
May 20 13:50:38.380025 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.379983    2554 generic.go:358] "Generic (PLEG): container finished" podID="d10bae7d-9cef-4c47-9918-5c9748ff88a2" containerID="5b3d6ccdede8010316166920f02909d3bc1fc9e73034d7098d95f616ae532924" exitCode=0
May 20 13:50:38.380142 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.380070    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerDied","Data":"5b3d6ccdede8010316166920f02909d3bc1fc9e73034d7098d95f616ae532924"}
May 20 13:50:38.381621 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.381537    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lh8dv" event={"ID":"f2b4c2c0-f318-4ceb-838e-a44c73eb275a","Type":"ContainerStarted","Data":"a0d76c7f366ae876942da9548256d7648d9b9aa7cad40da17189a9d575079659"}
May 20 13:50:38.382818 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.382798    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 13:50:38.384515 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.384494    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"a8b1c237ea48a1a259c3e34a1e5c4fc022c818948fbdaa6bd44f872526e8b187"}
May 20 13:50:38.384601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.384521    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"7085ad10fb1572b4416e4ca395dab412134dd4b7454f04b65a3f651d2852c5ed"}
May 20 13:50:38.384601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.384535    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"10e676380cc05162966a24c701d060194ebbfc7df204bc934a37f47a730a8089"}
May 20 13:50:38.384601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.384545    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"a2eff7f882a5754fb86a4a486ceeda75d4c3aaa7daf4e4bf8fa05b11c6d540aa"}
May 20 13:50:38.384601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.384555    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"dc8f996eed78632f54372872de210a461d39b36618abefeaab900670626b8964"}
May 20 13:50:38.384601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.384564    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"cc674901aaacefc86c908bcae3e5b0942b38ed092a8c1b7a664c5d1b269959dc"}
May 20 13:50:38.385896 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.385873    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6" event={"ID":"7e4366cf-e68e-4172-9240-ccdd09127623","Type":"ContainerStarted","Data":"35682159ad04ac79e0af265239e14e2c74bcc3a8812b786b2451e425deca6343"}
May 20 13:50:38.385985 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.385902    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6" event={"ID":"7e4366cf-e68e-4172-9240-ccdd09127623","Type":"ContainerStarted","Data":"f8347361b7213fcec392b5b931504e4f4ecb4007f67124c278fcca0a761179a4"}
May 20 13:50:38.387114 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.387094    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9nk9" event={"ID":"575a7241-3e3e-4fbd-a215-e69cb73bf5a4","Type":"ContainerStarted","Data":"c1687a0a3da4cfd74f79ada9b7c6315f0eb61ed72cb4bff11c516aaf29f3ca22"}
May 20 13:50:38.388272 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.388252    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fg6nm" event={"ID":"2790ca0e-0d38-4d36-94bd-f9d07e2aba29","Type":"ContainerStarted","Data":"526fa2a48b4e353598bd5f2885d4604afd6c16dabdfb3e02c374ce0b1d96b98d"}
May 20 13:50:38.421477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.421441    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:38.421646 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:38.421558    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:38.421646 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:38.421615    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret podName:5d5a03ba-819e-4e8c-b589-37e651c8f956 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.421597048 +0000 UTC m=+39.044322099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret") pod "global-pull-secret-syncer-smc4h" (UID: "5d5a03ba-819e-4e8c-b589-37e651c8f956") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:38.422627 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.422551    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-n9nk9" podStartSLOduration=3.774069384 podStartE2EDuration="22.42253939s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.627799696 +0000 UTC m=+3.250524745" lastFinishedPulling="2026-05-20 13:50:37.276269689 +0000 UTC m=+21.898994751" observedRunningTime="2026-05-20 13:50:38.422510754 +0000 UTC m=+23.045235823" watchObservedRunningTime="2026-05-20 13:50:38.42253939 +0000 UTC m=+23.045264457"
May 20 13:50:38.422747 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.422728    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k2wv5" podStartSLOduration=3.8644847540000002 podStartE2EDuration="22.422724601s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.631090379 +0000 UTC m=+3.253815438" lastFinishedPulling="2026-05-20 13:50:37.189330231 +0000 UTC m=+21.812055285" observedRunningTime="2026-05-20 13:50:38.40241341 +0000 UTC m=+23.025138641" watchObservedRunningTime="2026-05-20 13:50:38.422724601 +0000 UTC m=+23.045449707"
May 20 13:50:38.461550 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.461504    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-7cqn4" podStartSLOduration=3.896669283 podStartE2EDuration="22.46149129s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.624504226 +0000 UTC m=+3.247229276" lastFinishedPulling="2026-05-20 13:50:37.189326232 +0000 UTC m=+21.812051283" observedRunningTime="2026-05-20 13:50:38.461258171 +0000 UTC m=+23.083983238" watchObservedRunningTime="2026-05-20 13:50:38.46149129 +0000 UTC m=+23.084216358"
May 20 13:50:38.487252 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.487200    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fg6nm" podStartSLOduration=7.774688617 podStartE2EDuration="22.487187491s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.625711609 +0000 UTC m=+3.248436670" lastFinishedPulling="2026-05-20 13:50:33.338210493 +0000 UTC m=+17.960935544" observedRunningTime="2026-05-20 13:50:38.486765247 +0000 UTC m=+23.109490307" watchObservedRunningTime="2026-05-20 13:50:38.487187491 +0000 UTC m=+23.109912558"
May 20 13:50:38.514962 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:38.514914    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lh8dv" podStartSLOduration=3.9480250249999997 podStartE2EDuration="22.514900681s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.623684549 +0000 UTC m=+3.246409594" lastFinishedPulling="2026-05-20 13:50:37.19056019 +0000 UTC m=+21.813285250" observedRunningTime="2026-05-20 13:50:38.514417955 +0000 UTC m=+23.137143024" watchObservedRunningTime="2026-05-20 13:50:38.514900681 +0000 UTC m=+23.137625755"
May 20 13:50:39.120974 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:39.120871    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T13:50:38.382815228Z","UUID":"f0bd66eb-8286-4416-a42c-41c3b93f99c0","Handler":null,"Name":"","Endpoint":""}
May 20 13:50:39.123393 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:39.123367    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 13:50:39.123393 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:39.123397    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 13:50:39.392907 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:39.392537    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6" event={"ID":"7e4366cf-e68e-4172-9240-ccdd09127623","Type":"ContainerStarted","Data":"2956fa9b306a002176659ca7dda1b48286f361674c98ae151a096a5e2e5670b9"}
May 20 13:50:39.394205 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:39.394091    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5tbmv" event={"ID":"b78bf5cb-cabe-4493-9b8c-c3b0be0f493e","Type":"ContainerStarted","Data":"def527cbff2c63c3ddf9e321bf8f3b0465eaaa89abd6b24dda0b63e6aad2fa39"}
May 20 13:50:39.413402 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:39.413340    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2dn6" podStartSLOduration=2.787810817 podStartE2EDuration="23.413320933s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.632961305 +0000 UTC m=+3.255686352" lastFinishedPulling="2026-05-20 13:50:39.258471406 +0000 UTC m=+23.881196468" observedRunningTime="2026-05-20 13:50:39.412686173 +0000 UTC m=+24.035411241" watchObservedRunningTime="2026-05-20 13:50:39.413320933 +0000 UTC m=+24.036046002"
May 20 13:50:39.428165 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:39.428112    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5tbmv" podStartSLOduration=4.869187908 podStartE2EDuration="23.428095703s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.629977774 +0000 UTC m=+3.252702824" lastFinishedPulling="2026-05-20 13:50:37.188885569 +0000 UTC m=+21.811610619" observedRunningTime="2026-05-20 13:50:39.428055422 +0000 UTC m=+24.050780493" watchObservedRunningTime="2026-05-20 13:50:39.428095703 +0000 UTC m=+24.050820773"
May 20 13:50:40.172311 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:40.172275    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:40.172534 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:40.172275    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:40.172534 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:40.172404    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:40.172534 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:40.172276    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:40.172697 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:40.172560    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:40.172697 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:40.172650    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:40.399209 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:40.399168    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"66fa8f0dfd78eb132352ae0b4215132f45ea4eaf93a98dc34649210afee973e1"}
May 20 13:50:42.172531 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.172324    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:42.173260 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.172331    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:42.173260 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:42.172566    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:42.173260 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.172380    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:42.173260 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:42.172629    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:42.173260 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:42.172706    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:42.406743 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.406709    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" event={"ID":"32db493c-1a8f-4925-ada5-8f0562c64769","Type":"ContainerStarted","Data":"817012e92e6eac292548b6eb5b55270a5ba335a343bdaa247182f9e7b33a8131"}
May 20 13:50:42.407116 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.407093    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:42.407446 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.407124    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:42.423529 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.423430    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:42.434721 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.434664    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j" podStartSLOduration=7.843843174 podStartE2EDuration="26.434645729s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.622427937 +0000 UTC m=+3.245152982" lastFinishedPulling="2026-05-20 13:50:37.213230488 +0000 UTC m=+21.835955537" observedRunningTime="2026-05-20 13:50:42.433907983 +0000 UTC m=+27.056633052" watchObservedRunningTime="2026-05-20 13:50:42.434645729 +0000 UTC m=+27.057370798"
May 20 13:50:42.895857 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.895746    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:42.897038 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:42.897017    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:43.411882 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:43.411825    2554 generic.go:358] "Generic (PLEG): container finished" podID="d10bae7d-9cef-4c47-9918-5c9748ff88a2" containerID="2939fac2b6fe3575b48e8170cf871f68b6e4ee342ef6d12db3a3445b3b6ce971" exitCode=0
May 20 13:50:43.412355 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:43.411909    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerDied","Data":"2939fac2b6fe3575b48e8170cf871f68b6e4ee342ef6d12db3a3445b3b6ce971"}
May 20 13:50:43.412355 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:43.412082    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:43.412470 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:43.412431    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:43.413206 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:43.412670    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2wv5"
May 20 13:50:43.428969 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:43.428944    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:50:44.171894 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.171868    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:44.172003 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.171897    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:44.172003 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.171871    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:44.172003 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:44.171985    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:44.172087 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:44.172071    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:44.172246 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:44.172229    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:44.248343 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.248297    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z9p8j"]
May 20 13:50:44.248931 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.248902    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smc4h"]
May 20 13:50:44.250939 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.250910    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4cgv"]
May 20 13:50:44.415200 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.415168    2554 generic.go:358] "Generic (PLEG): container finished" podID="d10bae7d-9cef-4c47-9918-5c9748ff88a2" containerID="583e187ceffd3174694db47940d2dff4d599821348cdf91d66dddea1b91f9e1b" exitCode=0
May 20 13:50:44.415601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.415253    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerDied","Data":"583e187ceffd3174694db47940d2dff4d599821348cdf91d66dddea1b91f9e1b"}
May 20 13:50:44.415601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.415287    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:44.415601 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.415296    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:44.415601 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:44.415384    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:44.415601 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:44.415435    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:44.416215 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:44.416193    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:44.416329 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:44.416289    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:45.418607 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:45.418576    2554 generic.go:358] "Generic (PLEG): container finished" podID="d10bae7d-9cef-4c47-9918-5c9748ff88a2" containerID="b05b1acade4fe21806c290a14896b2466ca30c8c41c2683dffac41d47e46a067" exitCode=0
May 20 13:50:45.419016 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:45.418655    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerDied","Data":"b05b1acade4fe21806c290a14896b2466ca30c8c41c2683dffac41d47e46a067"}
May 20 13:50:46.173246 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:46.172991    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:46.173404 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:46.173067    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:46.173404 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:46.173341    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:46.173404 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:46.173095    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:46.173547 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:46.173426    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:46.173547 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:46.173508    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:48.171428 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:48.171386    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:48.172084 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:48.171386    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:48.172084 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:48.171539    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:48.172084 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:48.171579    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:48.172084 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:48.171386    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:48.172084 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:48.171667    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:49.705851 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:49.705810    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:49.706473 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:49.705959    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:49.706473 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:49.706023    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:51:21.706008012 +0000 UTC m=+66.328733062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:49.806210 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:49.806167    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:49.806398 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:49.806349    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:49.806398 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:49.806374    2554 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:49.806398 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:49.806387    2554 projected.go:194] Error preparing data for projected volume kube-api-access-tsr8m for pod openshift-network-diagnostics/network-check-target-z9p8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:49.806547 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:49.806454    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m podName:77da8e32-a72e-4ec4-96dc-97db3ede4765 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:21.806432407 +0000 UTC m=+66.429157463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tsr8m" (UniqueName: "kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m") pod "network-check-target-z9p8j" (UID: "77da8e32-a72e-4ec4-96dc-97db3ede4765") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:50.171937 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.171892    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:50.171937 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.171919    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:50.172173 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.171892    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:50.172173 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.172030    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z9p8j" podUID="77da8e32-a72e-4ec4-96dc-97db3ede4765"
May 20 13:50:50.172173 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.172103    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smc4h" podUID="5d5a03ba-819e-4e8c-b589-37e651c8f956"
May 20 13:50:50.172318 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.172199    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:50:50.183517 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.183497    2554 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-251.ec2.internal" event="NodeReady"
May 20 13:50:50.183683 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.183625    2554 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 13:50:50.223494 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.223458    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"]
May 20 13:50:50.254417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.254356    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"]
May 20 13:50:50.254417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.254402    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mtp2c"]
May 20 13:50:50.254666 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.254495    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.260943 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.260719    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 20 13:50:50.260943 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.260812    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 20 13:50:50.263254 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.262012    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 20 13:50:50.264944 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.264775    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gr7x7\""
May 20 13:50:50.268165 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.268143    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 20 13:50:50.270664 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.270640    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gngpj"]
May 20 13:50:50.270805 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.270786    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.273299 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.273277    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 13:50:50.273299 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.273295    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxfgl\""
May 20 13:50:50.273551 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.273533    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 13:50:50.292424 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.292398    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gngpj"]
May 20 13:50:50.292570 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.292509    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:50.292570 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.292510    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mtp2c"]
May 20 13:50:50.294990 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.294969    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 13:50:50.295598 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.295579    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 13:50:50.295728 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.295704    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxwgr\""
May 20 13:50:50.296249 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.296232    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 13:50:50.412221 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412186    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-trusted-ca\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412386 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412234    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412386 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412267    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-certificates\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412386 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412298    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-tmp-dir\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.412386 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412369    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.412580 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-bound-sa-token\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412580 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412442    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-image-registry-private-configuration\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412580 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412483    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:50.412580 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412524    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42wh\" (UniqueName: \"kubernetes.io/projected/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-kube-api-access-n42wh\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.412580 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412547    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2pv\" (UniqueName: \"kubernetes.io/projected/22742942-2c3a-4d4b-b354-8cf1476abc4e-kube-api-access-5t2pv\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:50.412784 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412583    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-ca-trust-extracted\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412784 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-installation-pull-secrets\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412784 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412655    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffbf\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-kube-api-access-jffbf\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.412784 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.412699    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-config-volume\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.514029 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.513936    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n42wh\" (UniqueName: \"kubernetes.io/projected/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-kube-api-access-n42wh\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.514029 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.513988    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2pv\" (UniqueName: \"kubernetes.io/projected/22742942-2c3a-4d4b-b354-8cf1476abc4e-kube-api-access-5t2pv\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514036    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-ca-trust-extracted\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514064    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-installation-pull-secrets\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514093    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jffbf\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-kube-api-access-jffbf\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514124    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-config-volume\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514161    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-trusted-ca\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514191    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514219    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-certificates\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514248    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-tmp-dir\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.514311    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.514331    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.514398    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:51.014376985 +0000 UTC m=+35.637102037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514499    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514543    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-bound-sa-token\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514591    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-tmp-dir\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514501    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-ca-trust-extracted\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514608    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-image-registry-private-configuration\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.514627    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:50.514640 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514646    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:50.515161 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.514683    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:51.014668819 +0000 UTC m=+35.637393870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:50:50.515161 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.514762    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:50.515161 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.514789    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-config-volume\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.515161 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:50.514823    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:50:51.014806884 +0000 UTC m=+35.637531935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:50:50.515400 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.515375    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-certificates\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.515532 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.515511    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-trusted-ca\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.520122 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.520090    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-installation-pull-secrets\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.520345 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.520324    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-image-registry-private-configuration\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.523571 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.523508    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-bound-sa-token\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:50.523687 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.523608    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2pv\" (UniqueName: \"kubernetes.io/projected/22742942-2c3a-4d4b-b354-8cf1476abc4e-kube-api-access-5t2pv\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:50.523752 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.523703    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42wh\" (UniqueName: \"kubernetes.io/projected/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-kube-api-access-n42wh\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:50.524082 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:50.524062    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffbf\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-kube-api-access-jffbf\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:51.018898 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:51.018829    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:51.018928    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:51.018963    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:51.018986    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:51.019052    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:51.019060    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:51.019076    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:51.019090    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:50:52.019056853 +0000 UTC m=+36.641781901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:51.019112    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:52.019102449 +0000 UTC m=+36.641827498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:50:51.019385 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:51.019124    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:52.019118068 +0000 UTC m=+36.641843115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:50:52.027808 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.027774    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.027828    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.027883    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:52.027942    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:52.027960    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:52.027971    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:52.027971    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:52.028019    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.028000217 +0000 UTC m=+38.650725263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:52.028036    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.028029804 +0000 UTC m=+38.650754850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:50:52.028205 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:52.028051    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.028045738 +0000 UTC m=+38.650770784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:50:52.171791 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.171588    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:50:52.172002 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.171598    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:50:52.172002 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.171613    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:52.176530 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.176503    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:50:52.176530 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.176530    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 13:50:52.176530 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.176535    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtwjf\""
May 20 13:50:52.176809 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.176579    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:50:52.176809 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.176597    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwf8m\""
May 20 13:50:52.176809 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.176535    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:50:52.436222 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.436191    2554 generic.go:358] "Generic (PLEG): container finished" podID="d10bae7d-9cef-4c47-9918-5c9748ff88a2" containerID="a4ee151716a6f1167cb8e33e267fae559a3cebc3a3359b1e4bbdd2c32c42d81f" exitCode=0
May 20 13:50:52.436390 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:52.436228    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerDied","Data":"a4ee151716a6f1167cb8e33e267fae559a3cebc3a3359b1e4bbdd2c32c42d81f"}
May 20 13:50:53.440365 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:53.440332    2554 generic.go:358] "Generic (PLEG): container finished" podID="d10bae7d-9cef-4c47-9918-5c9748ff88a2" containerID="611425132f2745ca4955a569ccf0a00fb2bc436f0c0469e3308d00c748ad4dbb" exitCode=0
May 20 13:50:53.440868 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:53.440383    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerDied","Data":"611425132f2745ca4955a569ccf0a00fb2bc436f0c0469e3308d00c748ad4dbb"}
May 20 13:50:54.043476 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.043386    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:54.043476 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.043458    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:54.043660 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.043516    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:54.043660 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:54.043543    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:54.043660 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:54.043594    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:50:54.043660 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:54.043606    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:50:54.043660 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:54.043627    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:50:58.043604992 +0000 UTC m=+42.666330055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:50:54.043875 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:54.043667    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:58.043660606 +0000 UTC m=+42.666385652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:50:54.043875 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:54.043604    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:54.043875 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:54.043736    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:50:58.043722368 +0000 UTC m=+42.666447413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:50:54.446005 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.445972    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:54.447938 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.447910    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-56jr7" event={"ID":"d10bae7d-9cef-4c47-9918-5c9748ff88a2","Type":"ContainerStarted","Data":"fd32f80a807726742c6e415cbe2918a373ec1b748eb937687c44b71a37c2074c"}
May 20 13:50:54.450117 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.450096    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d5a03ba-819e-4e8c-b589-37e651c8f956-original-pull-secret\") pod \"global-pull-secret-syncer-smc4h\" (UID: \"5d5a03ba-819e-4e8c-b589-37e651c8f956\") " pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:54.472095 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.472045    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-56jr7" podStartSLOduration=5.719469043 podStartE2EDuration="38.472027484s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:50:18.627259007 +0000 UTC m=+3.249984068" lastFinishedPulling="2026-05-20 13:50:51.37981746 +0000 UTC m=+36.002542509" observedRunningTime="2026-05-20 13:50:54.47038087 +0000 UTC m=+39.093105940" watchObservedRunningTime="2026-05-20 13:50:54.472027484 +0000 UTC m=+39.094752555"
May 20 13:50:54.594533 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.594494    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smc4h"
May 20 13:50:54.772550 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:54.772378    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smc4h"]
May 20 13:50:54.776318 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:50:54.776289    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5a03ba_819e_4e8c_b589_37e651c8f956.slice/crio-2fb788483b696928eafc6c5f2005f8bc9330d21d1e60bad94901b9588ee907f6 WatchSource:0}: Error finding container 2fb788483b696928eafc6c5f2005f8bc9330d21d1e60bad94901b9588ee907f6: Status 404 returned error can't find the container with id 2fb788483b696928eafc6c5f2005f8bc9330d21d1e60bad94901b9588ee907f6
May 20 13:50:55.450739 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:55.450690    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smc4h" event={"ID":"5d5a03ba-819e-4e8c-b589-37e651c8f956","Type":"ContainerStarted","Data":"2fb788483b696928eafc6c5f2005f8bc9330d21d1e60bad94901b9588ee907f6"}
May 20 13:50:58.076240 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:58.076190    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:58.076280    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:58.076300    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:58.076349    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:58.076372    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:51:06.076352442 +0000 UTC m=+50.699077504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:58.076441    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:58.076455    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:58.076504    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:06.076487812 +0000 UTC m=+50.699212861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:58.076441    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:58.076876 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:50:58.076579    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:51:06.076558363 +0000 UTC m=+50.699283411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:50:59.460300 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:59.460264    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smc4h" event={"ID":"5d5a03ba-819e-4e8c-b589-37e651c8f956","Type":"ContainerStarted","Data":"6fcc1ef793776224dd0509e4c27f683ad2ae6fa494eb2ef77530cc773770596b"}
May 20 13:50:59.475424 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:50:59.475375    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-smc4h" podStartSLOduration=33.557302221 podStartE2EDuration="37.475361906s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:54.777860439 +0000 UTC m=+39.400585488" lastFinishedPulling="2026-05-20 13:50:58.695920124 +0000 UTC m=+43.318645173" observedRunningTime="2026-05-20 13:50:59.475147834 +0000 UTC m=+44.097872902" watchObservedRunningTime="2026-05-20 13:50:59.475361906 +0000 UTC m=+44.098086974"
May 20 13:51:06.134298 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:06.134255    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:06.134325    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:06.134358    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:06.134400    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:06.134472    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:51:22.134456991 +0000 UTC m=+66.757182038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:06.134472    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:06.134475    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:06.134546    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:06.134527    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:51:22.13451429 +0000 UTC m=+66.757239337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:51:06.134914 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:06.134627    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:22.134612149 +0000 UTC m=+66.757337202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:51:15.429537 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:15.429501    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xd45j"
May 20 13:51:21.748567 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.748525    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:51:21.751264 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.751240    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:51:21.759242 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:21.759217    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 13:51:21.759329 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:21.759293    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:52:25.759274319 +0000 UTC m=+130.381999374 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : secret "metrics-daemon-secret" not found
May 20 13:51:21.849258 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.849226    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:51:21.851910 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.851893    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:51:21.862810 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.862790    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:51:21.872434 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.872406    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsr8m\" (UniqueName: \"kubernetes.io/projected/77da8e32-a72e-4ec4-96dc-97db3ede4765-kube-api-access-tsr8m\") pod \"network-check-target-z9p8j\" (UID: \"77da8e32-a72e-4ec4-96dc-97db3ede4765\") " pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:51:21.886852 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.886810    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtwjf\""
May 20 13:51:21.895009 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:21.894985    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:51:22.036806 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:22.036728    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z9p8j"]
May 20 13:51:22.039331 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:51:22.039303    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77da8e32_a72e_4ec4_96dc_97db3ede4765.slice/crio-53c586a408e85cac4d49a12d42fbcd0a81c14205755104394d1f4d2eb88dc22b WatchSource:0}: Error finding container 53c586a408e85cac4d49a12d42fbcd0a81c14205755104394d1f4d2eb88dc22b: Status 404 returned error can't find the container with id 53c586a408e85cac4d49a12d42fbcd0a81c14205755104394d1f4d2eb88dc22b
May 20 13:51:22.153418 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:22.153374    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:51:22.153418 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:22.153425    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:51:22.153671 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:22.153454    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:51:22.153671 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:22.153522    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:51:22.153671 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:22.153544    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:51:22.153671 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:22.153546    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:22.153671 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:22.153599    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:54.153583662 +0000 UTC m=+98.776308708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:51:22.153671 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:22.153614    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:51:54.153607414 +0000 UTC m=+98.776332460 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:51:22.153671 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:22.153635    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:22.153966 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:22.153733    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:51:54.153711987 +0000 UTC m=+98.776437051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:51:22.504557 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:22.504517    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z9p8j" event={"ID":"77da8e32-a72e-4ec4-96dc-97db3ede4765","Type":"ContainerStarted","Data":"53c586a408e85cac4d49a12d42fbcd0a81c14205755104394d1f4d2eb88dc22b"}
May 20 13:51:25.511266 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:25.511223    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z9p8j" event={"ID":"77da8e32-a72e-4ec4-96dc-97db3ede4765","Type":"ContainerStarted","Data":"0242c4fbf7f5f4bcef08bbf0ef313f2a91c61457059194958be6ccd17941847c"}
May 20 13:51:25.511715 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:25.511466    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:51:25.528310 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:25.528260    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z9p8j" podStartSLOduration=66.838426562 podStartE2EDuration="1m9.528245965s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:51:22.041375998 +0000 UTC m=+66.664101064" lastFinishedPulling="2026-05-20 13:51:24.731195421 +0000 UTC m=+69.353920467" observedRunningTime="2026-05-20 13:51:25.527632194 +0000 UTC m=+70.150357274" watchObservedRunningTime="2026-05-20 13:51:25.528245965 +0000 UTC m=+70.150971081"
May 20 13:51:54.189125 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:54.189083    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:54.189134    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:54.189162    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:54.189244    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:54.189253    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:54.189274    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:54.189275    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:54.189302    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:52:58.189288473 +0000 UTC m=+162.812013519 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:54.189339    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:58.189319386 +0000 UTC m=+162.812044432 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:51:54.189560 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:51:54.189354    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:52:58.189347841 +0000 UTC m=+162.812072887 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:51:56.516648 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:51:56.516620    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z9p8j"
May 20 13:52:25.818204 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:25.818157    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:52:25.818682 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:25.818311    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 13:52:25.818682 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:25.818383    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs podName:b8c29338-36d1-4d0d-942e-6b5312426efc nodeName:}" failed. No retries permitted until 2026-05-20 13:54:27.818367724 +0000 UTC m=+252.441092770 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs") pod "network-metrics-daemon-n4cgv" (UID: "b8c29338-36d1-4d0d-942e-6b5312426efc") : secret "metrics-daemon-secret" not found
May 20 13:52:45.393459 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.393423    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-8mb69"]
May 20 13:52:45.395263 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.395245    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.398042 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.398025    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 20 13:52:45.398237 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.398224    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 13:52:45.398871 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.398858    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tz4qk\""
May 20 13:52:45.399044 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.399033    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 20 13:52:45.399207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.399195    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 13:52:45.403091 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.403004    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 20 13:52:45.403870 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.403834    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-8mb69"]
May 20 13:52:45.495108 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.495059    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-kmw87"]
May 20 13:52:45.497146 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.497122    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.498904 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.498875    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"]
May 20 13:52:45.499678 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.499655    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 20 13:52:45.499783 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.499661    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 20 13:52:45.500493 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.500474    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l4v84\""
May 20 13:52:45.500661 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.500643    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 20 13:52:45.500721 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.500662    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.503594 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.503576    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 20 13:52:45.503688 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.503669    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:45.505216 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.505188    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 20 13:52:45.505216 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.505200    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 20 13:52:45.505435 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.505383    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:45.505505 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.505460    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-glkbb\""
May 20 13:52:45.514715 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.514685    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 20 13:52:45.515074 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.514804    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-kmw87"]
May 20 13:52:45.517519 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.517497    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"]
May 20 13:52:45.563782 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.563746    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e2746b-3239-4211-8013-f40b5b0f7ccc-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.563782 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.563785    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e2746b-3239-4211-8013-f40b5b0f7ccc-service-ca-bundle\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.564015 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.563808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qsb4\" (UniqueName: \"kubernetes.io/projected/61e2746b-3239-4211-8013-f40b5b0f7ccc-kube-api-access-4qsb4\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.564015 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.563860    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61e2746b-3239-4211-8013-f40b5b0f7ccc-tmp\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.564015 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.563902    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61e2746b-3239-4211-8013-f40b5b0f7ccc-serving-cert\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.564015 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.563945    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61e2746b-3239-4211-8013-f40b5b0f7ccc-snapshots\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.602165 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.602129    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"]
May 20 13:52:45.604100 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.604083    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"]
May 20 13:52:45.604252 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.604233    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"
May 20 13:52:45.605923 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.605897    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.606426 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.606406    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qwbbx\""
May 20 13:52:45.608051 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.608029    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 20 13:52:45.608146 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.608112    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:45.608146 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.608115    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 20 13:52:45.608254 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.608173    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gmr9x\""
May 20 13:52:45.608254 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.608144    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 20 13:52:45.614420 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.614399    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"]
May 20 13:52:45.615602 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.615578    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"]
May 20 13:52:45.665311 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665283    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/401a135b-3831-4cd9-944c-b230c3a84a91-config\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.665610 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665341    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e2746b-3239-4211-8013-f40b5b0f7ccc-service-ca-bundle\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.665610 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665365    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4qsb4\" (UniqueName: \"kubernetes.io/projected/61e2746b-3239-4211-8013-f40b5b0f7ccc-kube-api-access-4qsb4\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.665610 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665395    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75488ef6-b873-4e91-977f-d87376a79f00-serving-cert\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.665610 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665424    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/401a135b-3831-4cd9-944c-b230c3a84a91-serving-cert\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.665610 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665488    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61e2746b-3239-4211-8013-f40b5b0f7ccc-serving-cert\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.665610 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665511    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75488ef6-b873-4e91-977f-d87376a79f00-trusted-ca\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.665610 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665554    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61e2746b-3239-4211-8013-f40b5b0f7ccc-snapshots\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.666003 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh49j\" (UniqueName: \"kubernetes.io/projected/75488ef6-b873-4e91-977f-d87376a79f00-kube-api-access-jh49j\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.666003 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665705    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75488ef6-b873-4e91-977f-d87376a79f00-config\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.666003 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665736    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e2746b-3239-4211-8013-f40b5b0f7ccc-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.666003 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665871    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61e2746b-3239-4211-8013-f40b5b0f7ccc-tmp\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.666003 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.665907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n268s\" (UniqueName: \"kubernetes.io/projected/401a135b-3831-4cd9-944c-b230c3a84a91-kube-api-access-n268s\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.666250 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.666021    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e2746b-3239-4211-8013-f40b5b0f7ccc-service-ca-bundle\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.666305 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.666261    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/61e2746b-3239-4211-8013-f40b5b0f7ccc-snapshots\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.666385 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.666367    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61e2746b-3239-4211-8013-f40b5b0f7ccc-tmp\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.666532 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.666512    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e2746b-3239-4211-8013-f40b5b0f7ccc-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.667897 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.667881    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61e2746b-3239-4211-8013-f40b5b0f7ccc-serving-cert\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.673031 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.673007    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qsb4\" (UniqueName: \"kubernetes.io/projected/61e2746b-3239-4211-8013-f40b5b0f7ccc-kube-api-access-4qsb4\") pod \"insights-operator-78bdfb888c-8mb69\" (UID: \"61e2746b-3239-4211-8013-f40b5b0f7ccc\") " pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.704065 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.704030    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-8mb69"
May 20 13:52:45.767199 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767166    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n268s\" (UniqueName: \"kubernetes.io/projected/401a135b-3831-4cd9-944c-b230c3a84a91-kube-api-access-n268s\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.767362 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767220    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzscr\" (UniqueName: \"kubernetes.io/projected/e499f464-38b8-49aa-977c-c3b5054cd843-kube-api-access-dzscr\") pod \"network-check-source-7f44b6dd87-h5p4t\" (UID: \"e499f464-38b8-49aa-977c-c3b5054cd843\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"
May 20 13:52:45.767362 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-config\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.767362 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767298    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/401a135b-3831-4cd9-944c-b230c3a84a91-config\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.767523 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767381    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75488ef6-b873-4e91-977f-d87376a79f00-serving-cert\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.767523 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767422    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/401a135b-3831-4cd9-944c-b230c3a84a91-serving-cert\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.767523 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwfn\" (UniqueName: \"kubernetes.io/projected/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-kube-api-access-lwwfn\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.767523 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767502    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75488ef6-b873-4e91-977f-d87376a79f00-trusted-ca\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.767722 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767553    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.767722 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767604    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jh49j\" (UniqueName: \"kubernetes.io/projected/75488ef6-b873-4e91-977f-d87376a79f00-kube-api-access-jh49j\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.767722 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767638    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75488ef6-b873-4e91-977f-d87376a79f00-config\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.768047 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.767959    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/401a135b-3831-4cd9-944c-b230c3a84a91-config\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.768450 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.768421    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75488ef6-b873-4e91-977f-d87376a79f00-config\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.768813 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.768790    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75488ef6-b873-4e91-977f-d87376a79f00-trusted-ca\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.770443 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.770418    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/401a135b-3831-4cd9-944c-b230c3a84a91-serving-cert\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.771042 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.771024    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75488ef6-b873-4e91-977f-d87376a79f00-serving-cert\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.775351 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.775323    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh49j\" (UniqueName: \"kubernetes.io/projected/75488ef6-b873-4e91-977f-d87376a79f00-kube-api-access-jh49j\") pod \"console-operator-76dff48d59-kmw87\" (UID: \"75488ef6-b873-4e91-977f-d87376a79f00\") " pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.776525 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.776503    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n268s\" (UniqueName: \"kubernetes.io/projected/401a135b-3831-4cd9-944c-b230c3a84a91-kube-api-access-n268s\") pod \"service-ca-operator-57b5b68b8-9d68r\" (UID: \"401a135b-3831-4cd9-944c-b230c3a84a91\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.807501 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.807457    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:45.814315 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.814273    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"
May 20 13:52:45.829812 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.829771    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-8mb69"]
May 20 13:52:45.832561 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:52:45.832526    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e2746b_3239_4211_8013_f40b5b0f7ccc.slice/crio-707a44721bf6de0cc8cd85d34c11d00a40696f561d5cf2de1a37d948ac768127 WatchSource:0}: Error finding container 707a44721bf6de0cc8cd85d34c11d00a40696f561d5cf2de1a37d948ac768127: Status 404 returned error can't find the container with id 707a44721bf6de0cc8cd85d34c11d00a40696f561d5cf2de1a37d948ac768127
May 20 13:52:45.868420 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.868390    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.868533 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.868463    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dzscr\" (UniqueName: \"kubernetes.io/projected/e499f464-38b8-49aa-977c-c3b5054cd843-kube-api-access-dzscr\") pod \"network-check-source-7f44b6dd87-h5p4t\" (UID: \"e499f464-38b8-49aa-977c-c3b5054cd843\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"
May 20 13:52:45.868533 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.868498    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-config\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.869603 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.869260    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwfn\" (UniqueName: \"kubernetes.io/projected/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-kube-api-access-lwwfn\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.870306 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.870254    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-config\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.871673 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.871628    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.879787 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.879757    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwfn\" (UniqueName: \"kubernetes.io/projected/9a06524c-de8f-4f26-8ba4-ae198f93fdbf-kube-api-access-lwwfn\") pod \"kube-storage-version-migrator-operator-565644f49-mcdf2\" (UID: \"9a06524c-de8f-4f26-8ba4-ae198f93fdbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.880055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.880027    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzscr\" (UniqueName: \"kubernetes.io/projected/e499f464-38b8-49aa-977c-c3b5054cd843-kube-api-access-dzscr\") pod \"network-check-source-7f44b6dd87-h5p4t\" (UID: \"e499f464-38b8-49aa-977c-c3b5054cd843\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"
May 20 13:52:45.915147 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.915115    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"
May 20 13:52:45.920996 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.920937    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"
May 20 13:52:45.937589 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.937549    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-kmw87"]
May 20 13:52:45.942016 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:52:45.941982    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75488ef6_b873_4e91_977f_d87376a79f00.slice/crio-c9640f3cde28431ebb436ed6651b4d2c662ed09b5f541cda328772e59dcf8095 WatchSource:0}: Error finding container c9640f3cde28431ebb436ed6651b4d2c662ed09b5f541cda328772e59dcf8095: Status 404 returned error can't find the container with id c9640f3cde28431ebb436ed6651b4d2c662ed09b5f541cda328772e59dcf8095
May 20 13:52:45.960011 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:45.959952    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r"]
May 20 13:52:45.963786 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:52:45.963746    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401a135b_3831_4cd9_944c_b230c3a84a91.slice/crio-9313f818ab6f37c2ea732212f5ef316c490300d98370cb3f6b52ad15608e90af WatchSource:0}: Error finding container 9313f818ab6f37c2ea732212f5ef316c490300d98370cb3f6b52ad15608e90af: Status 404 returned error can't find the container with id 9313f818ab6f37c2ea732212f5ef316c490300d98370cb3f6b52ad15608e90af
May 20 13:52:46.055039 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.055008    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t"]
May 20 13:52:46.059362 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:52:46.059297    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode499f464_38b8_49aa_977c_c3b5054cd843.slice/crio-de42a8c991a4af8932abd63e461265f4e803d9ce5054bd8aadd4b417362435ad WatchSource:0}: Error finding container de42a8c991a4af8932abd63e461265f4e803d9ce5054bd8aadd4b417362435ad: Status 404 returned error can't find the container with id de42a8c991a4af8932abd63e461265f4e803d9ce5054bd8aadd4b417362435ad
May 20 13:52:46.068316 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.068290    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2"]
May 20 13:52:46.071883 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:52:46.071860    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a06524c_de8f_4f26_8ba4_ae198f93fdbf.slice/crio-857f102a8337d6bdc1090913135485e9b474f2a2dd6420911c036039557cbaa1 WatchSource:0}: Error finding container 857f102a8337d6bdc1090913135485e9b474f2a2dd6420911c036039557cbaa1: Status 404 returned error can't find the container with id 857f102a8337d6bdc1090913135485e9b474f2a2dd6420911c036039557cbaa1
May 20 13:52:46.673114 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.672975    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t" event={"ID":"e499f464-38b8-49aa-977c-c3b5054cd843","Type":"ContainerStarted","Data":"7e13a6487ed3e432185ec4b5ab2ac198f267f198f59a332050138084768820ba"}
May 20 13:52:46.673114 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.673021    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t" event={"ID":"e499f464-38b8-49aa-977c-c3b5054cd843","Type":"ContainerStarted","Data":"de42a8c991a4af8932abd63e461265f4e803d9ce5054bd8aadd4b417362435ad"}
May 20 13:52:46.675278 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.675169    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r" event={"ID":"401a135b-3831-4cd9-944c-b230c3a84a91","Type":"ContainerStarted","Data":"9313f818ab6f37c2ea732212f5ef316c490300d98370cb3f6b52ad15608e90af"}
May 20 13:52:46.678695 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.678645    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" event={"ID":"75488ef6-b873-4e91-977f-d87376a79f00","Type":"ContainerStarted","Data":"c9640f3cde28431ebb436ed6651b4d2c662ed09b5f541cda328772e59dcf8095"}
May 20 13:52:46.679811 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.679763    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8mb69" event={"ID":"61e2746b-3239-4211-8013-f40b5b0f7ccc","Type":"ContainerStarted","Data":"707a44721bf6de0cc8cd85d34c11d00a40696f561d5cf2de1a37d948ac768127"}
May 20 13:52:46.680871 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.680806    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2" event={"ID":"9a06524c-de8f-4f26-8ba4-ae198f93fdbf","Type":"ContainerStarted","Data":"857f102a8337d6bdc1090913135485e9b474f2a2dd6420911c036039557cbaa1"}
May 20 13:52:46.690109 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:46.689995    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-h5p4t" podStartSLOduration=1.689980711 podStartE2EDuration="1.689980711s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:46.689519415 +0000 UTC m=+151.312244484" watchObservedRunningTime="2026-05-20 13:52:46.689980711 +0000 UTC m=+151.312705779"
May 20 13:52:47.858416 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.858376    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"]
May 20 13:52:47.861020 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.860991    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:47.863517 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.863492    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 20 13:52:47.863517 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.863506    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hsmpm\""
May 20 13:52:47.863687 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.863498    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 20 13:52:47.870756 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.870727    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"]
May 20 13:52:47.988230 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.988190    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a65ae7e-362a-42cd-a0d2-8275da7461f9-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:47.988393 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:47.988252    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:48.089458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:48.089415    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a65ae7e-362a-42cd-a0d2-8275da7461f9-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:48.089658 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:48.089473    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:48.089658 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:48.089630    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 13:52:48.089777 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:48.089697    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert podName:7a65ae7e-362a-42cd-a0d2-8275da7461f9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:48.589677286 +0000 UTC m=+153.212402384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-rcxc9" (UID: "7a65ae7e-362a-42cd-a0d2-8275da7461f9") : secret "networking-console-plugin-cert" not found
May 20 13:52:48.090314 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:48.090269    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a65ae7e-362a-42cd-a0d2-8275da7461f9-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:48.593783 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:48.593747    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:48.593978 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:48.593950    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 13:52:48.594046 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:48.594035    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert podName:7a65ae7e-362a-42cd-a0d2-8275da7461f9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:49.594012301 +0000 UTC m=+154.216737369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-rcxc9" (UID: "7a65ae7e-362a-42cd-a0d2-8275da7461f9") : secret "networking-console-plugin-cert" not found
May 20 13:52:49.604223 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:49.604183    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:49.604647 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:49.604357    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 13:52:49.604647 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:49.604439    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert podName:7a65ae7e-362a-42cd-a0d2-8275da7461f9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:51.604417667 +0000 UTC m=+156.227142716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-rcxc9" (UID: "7a65ae7e-362a-42cd-a0d2-8275da7461f9") : secret "networking-console-plugin-cert" not found
May 20 13:52:50.696949 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.696912    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r" event={"ID":"401a135b-3831-4cd9-944c-b230c3a84a91","Type":"ContainerStarted","Data":"f801d55d3f6d3c600e22015facef0cd4b10d70651d2a338a8440b0e635537f29"}
May 20 13:52:50.698485 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.698466    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/0.log"
May 20 13:52:50.698624 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.698502    2554 generic.go:358] "Generic (PLEG): container finished" podID="75488ef6-b873-4e91-977f-d87376a79f00" containerID="22108a4e3a5ed6672cc44c1b4d0b91f6b84dc88e9c95c991bba5354274a981aa" exitCode=255
May 20 13:52:50.698624 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.698570    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" event={"ID":"75488ef6-b873-4e91-977f-d87376a79f00","Type":"ContainerDied","Data":"22108a4e3a5ed6672cc44c1b4d0b91f6b84dc88e9c95c991bba5354274a981aa"}
May 20 13:52:50.698814 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.698793    2554 scope.go:117] "RemoveContainer" containerID="22108a4e3a5ed6672cc44c1b4d0b91f6b84dc88e9c95c991bba5354274a981aa"
May 20 13:52:50.700255 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.700229    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8mb69" event={"ID":"61e2746b-3239-4211-8013-f40b5b0f7ccc","Type":"ContainerStarted","Data":"030fb2707063845779da55e522e9a33abda066f0ff863e9192100a30928c1379"}
May 20 13:52:50.701565 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.701531    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2" event={"ID":"9a06524c-de8f-4f26-8ba4-ae198f93fdbf","Type":"ContainerStarted","Data":"4ae8bad1ee6110f97dc03d154ba7a82d9259cf8edf43c5b1d750a60be11887b7"}
May 20 13:52:50.737517 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.737449    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r" podStartSLOduration=2.081691627 podStartE2EDuration="5.737432527s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="2026-05-20 13:52:45.965619879 +0000 UTC m=+150.588344925" lastFinishedPulling="2026-05-20 13:52:49.621360778 +0000 UTC m=+154.244085825" observedRunningTime="2026-05-20 13:52:50.737097579 +0000 UTC m=+155.359822647" watchObservedRunningTime="2026-05-20 13:52:50.737432527 +0000 UTC m=+155.360157596"
May 20 13:52:50.785514 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.785481    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"]
May 20 13:52:50.789078 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.789055    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"
May 20 13:52:50.790819 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.790680    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2" podStartSLOduration=2.240352423 podStartE2EDuration="5.790663079s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="2026-05-20 13:52:46.073680356 +0000 UTC m=+150.696405405" lastFinishedPulling="2026-05-20 13:52:49.623991 +0000 UTC m=+154.246716061" observedRunningTime="2026-05-20 13:52:50.790409374 +0000 UTC m=+155.413134442" watchObservedRunningTime="2026-05-20 13:52:50.790663079 +0000 UTC m=+155.413388148"
May 20 13:52:50.791820 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.791627    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 20 13:52:50.791820 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.791728    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tvh9t\""
May 20 13:52:50.792027 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.791987    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 20 13:52:50.798825 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.798799    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"]
May 20 13:52:50.814917 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.814381    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-8mb69" podStartSLOduration=2.030537837 podStartE2EDuration="5.814361898s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="2026-05-20 13:52:45.834383091 +0000 UTC m=+150.457108137" lastFinishedPulling="2026-05-20 13:52:49.618207136 +0000 UTC m=+154.240932198" observedRunningTime="2026-05-20 13:52:50.814159416 +0000 UTC m=+155.436884487" watchObservedRunningTime="2026-05-20 13:52:50.814361898 +0000 UTC m=+155.437086973"
May 20 13:52:50.817795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.815960    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96sj\" (UniqueName: \"kubernetes.io/projected/216fb31e-13ab-4026-a416-7848a037a823-kube-api-access-m96sj\") pod \"migrator-b985f5d89-fq9s5\" (UID: \"216fb31e-13ab-4026-a416-7848a037a823\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"
May 20 13:52:50.916711 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.916671    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m96sj\" (UniqueName: \"kubernetes.io/projected/216fb31e-13ab-4026-a416-7848a037a823-kube-api-access-m96sj\") pod \"migrator-b985f5d89-fq9s5\" (UID: \"216fb31e-13ab-4026-a416-7848a037a823\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"
May 20 13:52:50.925006 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:50.924978    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96sj\" (UniqueName: \"kubernetes.io/projected/216fb31e-13ab-4026-a416-7848a037a823-kube-api-access-m96sj\") pod \"migrator-b985f5d89-fq9s5\" (UID: \"216fb31e-13ab-4026-a416-7848a037a823\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"
May 20 13:52:51.103439 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.103349    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"
May 20 13:52:51.236379 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.236348    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5"]
May 20 13:52:51.241744 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:52:51.241712    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216fb31e_13ab_4026_a416_7848a037a823.slice/crio-349e7dac0cb1c886cf614252b1547ec8dbcd1bb20b903e9b24a88280151e06e5 WatchSource:0}: Error finding container 349e7dac0cb1c886cf614252b1547ec8dbcd1bb20b903e9b24a88280151e06e5: Status 404 returned error can't find the container with id 349e7dac0cb1c886cf614252b1547ec8dbcd1bb20b903e9b24a88280151e06e5
May 20 13:52:51.622203 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.622170    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:51.622370 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:51.622291    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 13:52:51.622370 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:51.622347    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert podName:7a65ae7e-362a-42cd-a0d2-8275da7461f9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:55.622331292 +0000 UTC m=+160.245056338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-rcxc9" (UID: "7a65ae7e-362a-42cd-a0d2-8275da7461f9") : secret "networking-console-plugin-cert" not found
May 20 13:52:51.705938 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.705910    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 13:52:51.706346 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.706265    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/0.log"
May 20 13:52:51.706346 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.706299    2554 generic.go:358] "Generic (PLEG): container finished" podID="75488ef6-b873-4e91-977f-d87376a79f00" containerID="f8d6ceca89ef779529e5bea6708d6080a0ee86204ac4d1a27e3d8c99d7ecbd3b" exitCode=255
May 20 13:52:51.706443 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.706371    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" event={"ID":"75488ef6-b873-4e91-977f-d87376a79f00","Type":"ContainerDied","Data":"f8d6ceca89ef779529e5bea6708d6080a0ee86204ac4d1a27e3d8c99d7ecbd3b"}
May 20 13:52:51.706443 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.706429    2554 scope.go:117] "RemoveContainer" containerID="22108a4e3a5ed6672cc44c1b4d0b91f6b84dc88e9c95c991bba5354274a981aa"
May 20 13:52:51.706727 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.706708    2554 scope.go:117] "RemoveContainer" containerID="f8d6ceca89ef779529e5bea6708d6080a0ee86204ac4d1a27e3d8c99d7ecbd3b"
May 20 13:52:51.706954 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:51.706934    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-kmw87_openshift-console-operator(75488ef6-b873-4e91-977f-d87376a79f00)\"" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" podUID="75488ef6-b873-4e91-977f-d87376a79f00"
May 20 13:52:51.707654 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:51.707478    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5" event={"ID":"216fb31e-13ab-4026-a416-7848a037a823","Type":"ContainerStarted","Data":"349e7dac0cb1c886cf614252b1547ec8dbcd1bb20b903e9b24a88280151e06e5"}
May 20 13:52:52.715058 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:52.714976    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 13:52:52.715471 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:52.715372    2554 scope.go:117] "RemoveContainer" containerID="f8d6ceca89ef779529e5bea6708d6080a0ee86204ac4d1a27e3d8c99d7ecbd3b"
May 20 13:52:52.715592 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:52.715572    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-kmw87_openshift-console-operator(75488ef6-b873-4e91-977f-d87376a79f00)\"" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" podUID="75488ef6-b873-4e91-977f-d87376a79f00"
May 20 13:52:52.716589 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:52.716558    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5" event={"ID":"216fb31e-13ab-4026-a416-7848a037a823","Type":"ContainerStarted","Data":"6f4310699b93528f012f0d5636f5cb2009adeba5da5cdcb5a2f652b765d3aade"}
May 20 13:52:52.716589 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:52.716585    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5" event={"ID":"216fb31e-13ab-4026-a416-7848a037a823","Type":"ContainerStarted","Data":"40bde2dcab65679d6ba6d1ee1e16417ea89dc1de9e91b57b8f46154caed15204"}
May 20 13:52:52.744375 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:52.744329    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-fq9s5" podStartSLOduration=1.567734792 podStartE2EDuration="2.744314736s" podCreationTimestamp="2026-05-20 13:52:50 +0000 UTC" firstStartedPulling="2026-05-20 13:52:51.244646295 +0000 UTC m=+155.867371344" lastFinishedPulling="2026-05-20 13:52:52.421226232 +0000 UTC m=+157.043951288" observedRunningTime="2026-05-20 13:52:52.743417848 +0000 UTC m=+157.366142927" watchObservedRunningTime="2026-05-20 13:52:52.744314736 +0000 UTC m=+157.367039804"
May 20 13:52:53.270468 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:53.270419    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt" podUID="da6f1cd9-dbc0-445c-a6d4-dc3497d10893"
May 20 13:52:53.279659 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:53.279636    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mtp2c" podUID="17a72c68-134e-4e0e-a8e7-e278d52ef7dc"
May 20 13:52:53.302071 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:53.302015    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-gngpj" podUID="22742942-2c3a-4d4b-b354-8cf1476abc4e"
May 20 13:52:53.719813 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:53.719782    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtp2c"
May 20 13:52:53.720289 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:53.719782    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:52:53.720289 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:53.719782    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:52:54.146603 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.146530    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7cqn4_f817fca4-539b-4c30-8739-dd4417db0223/dns-node-resolver/0.log"
May 20 13:52:54.707596 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.707563    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-7rxlv"]
May 20 13:52:54.710536 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.710520    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.713163 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.713132    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 20 13:52:54.713296 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.713181    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 20 13:52:54.713296 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.713240    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zjr8q\""
May 20 13:52:54.714039 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.714024    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 20 13:52:54.714098 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.714066    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 20 13:52:54.720822 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.720774    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-7rxlv"]
May 20 13:52:54.747322 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.747285    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/adf89ccf-bc26-4d82-b557-84ce4420b2ae-signing-key\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.747472 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.747333    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/adf89ccf-bc26-4d82-b557-84ce4420b2ae-signing-cabundle\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.747472 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.747427    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbx5t\" (UniqueName: \"kubernetes.io/projected/adf89ccf-bc26-4d82-b557-84ce4420b2ae-kube-api-access-kbx5t\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.848899 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.848830    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/adf89ccf-bc26-4d82-b557-84ce4420b2ae-signing-key\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.848899 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.848903    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/adf89ccf-bc26-4d82-b557-84ce4420b2ae-signing-cabundle\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.849154 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.848944    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kbx5t\" (UniqueName: \"kubernetes.io/projected/adf89ccf-bc26-4d82-b557-84ce4420b2ae-kube-api-access-kbx5t\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.849644 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.849622    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/adf89ccf-bc26-4d82-b557-84ce4420b2ae-signing-cabundle\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.851410 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.851382    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/adf89ccf-bc26-4d82-b557-84ce4420b2ae-signing-key\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:54.857672 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:54.857649    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbx5t\" (UniqueName: \"kubernetes.io/projected/adf89ccf-bc26-4d82-b557-84ce4420b2ae-kube-api-access-kbx5t\") pod \"service-ca-67d667c7d-7rxlv\" (UID: \"adf89ccf-bc26-4d82-b557-84ce4420b2ae\") " pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:55.019949 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.019835    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-7rxlv"
May 20 13:52:55.146198 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.146167    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-7rxlv"]
May 20 13:52:55.149005 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:52:55.148969    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf89ccf_bc26_4d82_b557_84ce4420b2ae.slice/crio-4db0bfa5bc20435d4a2c361a0b83d188fbb4f6b7845fe56c438ccc65d57b5228 WatchSource:0}: Error finding container 4db0bfa5bc20435d4a2c361a0b83d188fbb4f6b7845fe56c438ccc65d57b5228: Status 404 returned error can't find the container with id 4db0bfa5bc20435d4a2c361a0b83d188fbb4f6b7845fe56c438ccc65d57b5228
May 20 13:52:55.190829 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:55.190797    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-n4cgv" podUID="b8c29338-36d1-4d0d-942e-6b5312426efc"
May 20 13:52:55.349361 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.349282    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fg6nm_2790ca0e-0d38-4d36-94bd-f9d07e2aba29/node-ca/0.log"
May 20 13:52:55.656466 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.656433    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:52:55.656643 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:55.656586    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 13:52:55.656698 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:55.656652    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert podName:7a65ae7e-362a-42cd-a0d2-8275da7461f9 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:03.65663655 +0000 UTC m=+168.279361596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-rcxc9" (UID: "7a65ae7e-362a-42cd-a0d2-8275da7461f9") : secret "networking-console-plugin-cert" not found
May 20 13:52:55.725751 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.725707    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-7rxlv" event={"ID":"adf89ccf-bc26-4d82-b557-84ce4420b2ae","Type":"ContainerStarted","Data":"4121bba4d69d3547faa193cc97a4959aacfbd9ffee36f10da37bb58251db7edd"}
May 20 13:52:55.725751 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.725755    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-7rxlv" event={"ID":"adf89ccf-bc26-4d82-b557-84ce4420b2ae","Type":"ContainerStarted","Data":"4db0bfa5bc20435d4a2c361a0b83d188fbb4f6b7845fe56c438ccc65d57b5228"}
May 20 13:52:55.744011 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.743959    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-7rxlv" podStartSLOduration=1.7439418469999999 podStartE2EDuration="1.743941847s" podCreationTimestamp="2026-05-20 13:52:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:55.742877099 +0000 UTC m=+160.365602167" watchObservedRunningTime="2026-05-20 13:52:55.743941847 +0000 UTC m=+160.366666952"
May 20 13:52:55.808170 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.808130    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:55.808170 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.808171    2554 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:52:55.808611 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:55.808593    2554 scope.go:117] "RemoveContainer" containerID="f8d6ceca89ef779529e5bea6708d6080a0ee86204ac4d1a27e3d8c99d7ecbd3b"
May 20 13:52:55.808805 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:55.808784    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-kmw87_openshift-console-operator(75488ef6-b873-4e91-977f-d87376a79f00)\"" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" podUID="75488ef6-b873-4e91-977f-d87376a79f00"
May 20 13:52:56.548709 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:56.548671    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fq9s5_216fb31e-13ab-4026-a416-7848a037a823/migrator/0.log"
May 20 13:52:56.748030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:56.747977    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fq9s5_216fb31e-13ab-4026-a416-7848a037a823/graceful-termination/0.log"
May 20 13:52:56.948631 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:56.948586    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mcdf2_9a06524c-de8f-4f26-8ba4-ae198f93fdbf/kube-storage-version-migrator-operator/0.log"
May 20 13:52:58.279002 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:58.278963    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:58.279052    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") pod \"image-registry-7dfc8fbf59-4zfzt\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") " pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:52:58.279088    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:58.279109    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:58.279167    2554 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:58.279181    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls podName:17a72c68-134e-4e0e-a8e7-e278d52ef7dc nodeName:}" failed. No retries permitted until 2026-05-20 13:55:00.279165179 +0000 UTC m=+284.901890226 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls") pod "dns-default-mtp2c" (UID: "17a72c68-134e-4e0e-a8e7-e278d52ef7dc") : secret "dns-default-metrics-tls" not found
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:58.279196    2554 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:58.279214    2554 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dfc8fbf59-4zfzt: secret "image-registry-tls" not found
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:58.279205    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert podName:22742942-2c3a-4d4b-b354-8cf1476abc4e nodeName:}" failed. No retries permitted until 2026-05-20 13:55:00.279194182 +0000 UTC m=+284.901919233 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert") pod "ingress-canary-gngpj" (UID: "22742942-2c3a-4d4b-b354-8cf1476abc4e") : secret "canary-serving-cert" not found
May 20 13:52:58.279450 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:52:58.279280    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls podName:da6f1cd9-dbc0-445c-a6d4-dc3497d10893 nodeName:}" failed. No retries permitted until 2026-05-20 13:55:00.279257361 +0000 UTC m=+284.901982425 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls") pod "image-registry-7dfc8fbf59-4zfzt" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893") : secret "image-registry-tls" not found
May 20 13:53:03.725642 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:03.725601    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:53:03.726143 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:03.725774    2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 20 13:53:03.726143 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:03.725880    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert podName:7a65ae7e-362a-42cd-a0d2-8275da7461f9 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:19.7258572 +0000 UTC m=+184.348582248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-rcxc9" (UID: "7a65ae7e-362a-42cd-a0d2-8275da7461f9") : secret "networking-console-plugin-cert" not found
May 20 13:53:07.172038 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:07.171963    2554 scope.go:117] "RemoveContainer" containerID="f8d6ceca89ef779529e5bea6708d6080a0ee86204ac4d1a27e3d8c99d7ecbd3b"
May 20 13:53:07.763695 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:07.763660    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 13:53:07.763891 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:07.763733    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" event={"ID":"75488ef6-b873-4e91-977f-d87376a79f00","Type":"ContainerStarted","Data":"ad9f857b3ea2e9cf50736c6437dd7c8c2ee139b90f6100f14a77d0204c844a6c"}
May 20 13:53:07.764089 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:07.764066    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:53:07.771771 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:07.771737    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-kmw87"
May 20 13:53:07.780857 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:07.780798    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-kmw87" podStartSLOduration=19.106735203 podStartE2EDuration="22.780783522s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="2026-05-20 13:52:45.944160696 +0000 UTC m=+150.566885746" lastFinishedPulling="2026-05-20 13:52:49.618209006 +0000 UTC m=+154.240934065" observedRunningTime="2026-05-20 13:53:07.780231093 +0000 UTC m=+172.402956169" watchObservedRunningTime="2026-05-20 13:53:07.780783522 +0000 UTC m=+172.403508623"
May 20 13:53:08.172276 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:08.172239    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:53:17.231460 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.231424    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f49c5"]
May 20 13:53:17.234499 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.234480    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.240764 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.240738    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 13:53:17.241808 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.241790    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tgnrd\""
May 20 13:53:17.247961 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.247944    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 13:53:17.267060 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.267017    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f49c5"]
May 20 13:53:17.334925 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.334884    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.335110 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.334940    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.335110 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.334990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-crio-socket\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.335110 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.335026    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-data-volume\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.335110 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.335066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkc7g\" (UniqueName: \"kubernetes.io/projected/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-kube-api-access-dkc7g\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436053 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436016    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436223 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436061    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436223 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436084    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-crio-socket\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436223 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436102    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-data-volume\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436223 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436122    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dkc7g\" (UniqueName: \"kubernetes.io/projected/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-kube-api-access-dkc7g\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436379 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436220    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-crio-socket\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436506 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436473    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-data-volume\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.436625 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.436547    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.438406 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.438381    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.461492 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.461453    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkc7g\" (UniqueName: \"kubernetes.io/projected/54a00923-9d3c-4d5c-98cf-adbbb391b2ca-kube-api-access-dkc7g\") pod \"insights-runtime-extractor-f49c5\" (UID: \"54a00923-9d3c-4d5c-98cf-adbbb391b2ca\") " pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.542896 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.542772    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f49c5"
May 20 13:53:17.683503 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.683463    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f49c5"]
May 20 13:53:17.686441 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:17.686407    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a00923_9d3c_4d5c_98cf_adbbb391b2ca.slice/crio-e515aa96462646288ecf2d379363d77ae58319dbd2d54a5017b731d5b12954ec WatchSource:0}: Error finding container e515aa96462646288ecf2d379363d77ae58319dbd2d54a5017b731d5b12954ec: Status 404 returned error can't find the container with id e515aa96462646288ecf2d379363d77ae58319dbd2d54a5017b731d5b12954ec
May 20 13:53:17.789353 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.789314    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f49c5" event={"ID":"54a00923-9d3c-4d5c-98cf-adbbb391b2ca","Type":"ContainerStarted","Data":"e507a24971e46450193912257fb78a094a8449be4c8c4acdb097547cbbc8260c"}
May 20 13:53:17.789353 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:17.789351    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f49c5" event={"ID":"54a00923-9d3c-4d5c-98cf-adbbb391b2ca","Type":"ContainerStarted","Data":"e515aa96462646288ecf2d379363d77ae58319dbd2d54a5017b731d5b12954ec"}
May 20 13:53:18.793772 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:18.793737    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f49c5" event={"ID":"54a00923-9d3c-4d5c-98cf-adbbb391b2ca","Type":"ContainerStarted","Data":"9fbb30d67e499d978aff0683619ccd422c643efcc9b9b3484cc326b6d1376563"}
May 20 13:53:19.755818 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:19.755774    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:53:19.758358 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:19.758331    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a65ae7e-362a-42cd-a0d2-8275da7461f9-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-rcxc9\" (UID: \"7a65ae7e-362a-42cd-a0d2-8275da7461f9\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:53:19.798037 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:19.798001    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f49c5" event={"ID":"54a00923-9d3c-4d5c-98cf-adbbb391b2ca","Type":"ContainerStarted","Data":"bba0eb5fa90c23cdc783b00c9edc6548187e88e43ab68d6361cddf5145f4ddd0"}
May 20 13:53:19.816492 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:19.816433    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f49c5" podStartSLOduration=0.911597249 podStartE2EDuration="2.816418981s" podCreationTimestamp="2026-05-20 13:53:17 +0000 UTC" firstStartedPulling="2026-05-20 13:53:17.734803088 +0000 UTC m=+182.357528138" lastFinishedPulling="2026-05-20 13:53:19.639624823 +0000 UTC m=+184.262349870" observedRunningTime="2026-05-20 13:53:19.816014679 +0000 UTC m=+184.438739767" watchObservedRunningTime="2026-05-20 13:53:19.816418981 +0000 UTC m=+184.439144049"
May 20 13:53:19.975030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:19.974993    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hsmpm\""
May 20 13:53:19.982747 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:19.982723    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"
May 20 13:53:20.118801 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.118764    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9"]
May 20 13:53:20.121996 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:20.121970    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a65ae7e_362a_42cd_a0d2_8275da7461f9.slice/crio-f6be40faf50fa0ec24a12747f0467457e59965d518658079e06a672e1eb9ba57 WatchSource:0}: Error finding container f6be40faf50fa0ec24a12747f0467457e59965d518658079e06a672e1eb9ba57: Status 404 returned error can't find the container with id f6be40faf50fa0ec24a12747f0467457e59965d518658079e06a672e1eb9ba57
May 20 13:53:20.572498 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.572461    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"]
May 20 13:53:20.575430 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.575407    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"
May 20 13:53:20.577862 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.577827    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 20 13:53:20.577973 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.577896    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pvkq8\""
May 20 13:53:20.583965 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.583942    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"]
May 20 13:53:20.663647 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.663609    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e9938334-d9bc-489a-ba96-685498e8af9c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2\" (UID: \"e9938334-d9bc-489a-ba96-685498e8af9c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"
May 20 13:53:20.764382 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.764331    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e9938334-d9bc-489a-ba96-685498e8af9c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2\" (UID: \"e9938334-d9bc-489a-ba96-685498e8af9c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"
May 20 13:53:20.767293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.767264    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e9938334-d9bc-489a-ba96-685498e8af9c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2\" (UID: \"e9938334-d9bc-489a-ba96-685498e8af9c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"
May 20 13:53:20.801611 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.801563    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9" event={"ID":"7a65ae7e-362a-42cd-a0d2-8275da7461f9","Type":"ContainerStarted","Data":"f6be40faf50fa0ec24a12747f0467457e59965d518658079e06a672e1eb9ba57"}
May 20 13:53:20.885029 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:20.884942    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"
May 20 13:53:21.261441 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:21.261412    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"]
May 20 13:53:21.264360 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:21.264335    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9938334_d9bc_489a_ba96_685498e8af9c.slice/crio-61d078e65eb28798a5e4b54ce826cc293ee1f13af19edaaf5ba684e6460fce10 WatchSource:0}: Error finding container 61d078e65eb28798a5e4b54ce826cc293ee1f13af19edaaf5ba684e6460fce10: Status 404 returned error can't find the container with id 61d078e65eb28798a5e4b54ce826cc293ee1f13af19edaaf5ba684e6460fce10
May 20 13:53:21.805934 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:21.805894    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9" event={"ID":"7a65ae7e-362a-42cd-a0d2-8275da7461f9","Type":"ContainerStarted","Data":"ee6beae00e873711adbf92c45117023bc7589917fed38171abbc063251b5c272"}
May 20 13:53:21.807141 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:21.807105    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2" event={"ID":"e9938334-d9bc-489a-ba96-685498e8af9c","Type":"ContainerStarted","Data":"61d078e65eb28798a5e4b54ce826cc293ee1f13af19edaaf5ba684e6460fce10"}
May 20 13:53:22.810881 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:22.810828    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2" event={"ID":"e9938334-d9bc-489a-ba96-685498e8af9c","Type":"ContainerStarted","Data":"9bd1a282d5479ec5aca8ca946f47fdfb385f10a10ac97886cf1d0e61c9de36c2"}
May 20 13:53:22.828293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:22.828244    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-rcxc9" podStartSLOduration=34.783074594 podStartE2EDuration="35.828227471s" podCreationTimestamp="2026-05-20 13:52:47 +0000 UTC" firstStartedPulling="2026-05-20 13:53:20.123791599 +0000 UTC m=+184.746516645" lastFinishedPulling="2026-05-20 13:53:21.168944464 +0000 UTC m=+185.791669522" observedRunningTime="2026-05-20 13:53:21.821625 +0000 UTC m=+186.444350066" watchObservedRunningTime="2026-05-20 13:53:22.828227471 +0000 UTC m=+187.450952538"
May 20 13:53:22.828446 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:22.828323    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2" podStartSLOduration=1.848773273 podStartE2EDuration="2.828318434s" podCreationTimestamp="2026-05-20 13:53:20 +0000 UTC" firstStartedPulling="2026-05-20 13:53:21.266254173 +0000 UTC m=+185.888979222" lastFinishedPulling="2026-05-20 13:53:22.245799334 +0000 UTC m=+186.868524383" observedRunningTime="2026-05-20 13:53:22.826358668 +0000 UTC m=+187.449083736" watchObservedRunningTime="2026-05-20 13:53:22.828318434 +0000 UTC m=+187.451043510"
May 20 13:53:23.815742 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:23.815698    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"
May 20 13:53:23.820500 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:23.820475    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2"
May 20 13:53:26.045901 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.045867    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c94b6569-bpvw8"]
May 20 13:53:26.048195 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.048166    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.050580 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.050556    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 20 13:53:26.051548 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.051387    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 20 13:53:26.051548 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.051393    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dc7gl\""
May 20 13:53:26.051548 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.051448    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 20 13:53:26.051548 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.051467    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 20 13:53:26.051548 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.051496    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 20 13:53:26.051548 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.051506    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 20 13:53:26.052029 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.051624    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 20 13:53:26.058599 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.058575    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c94b6569-bpvw8"]
May 20 13:53:26.210831 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.210793    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-serving-cert\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.211030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.210860    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-console-config\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.211030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.210926    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-oauth-serving-cert\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.211030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.210955    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrc2x\" (UniqueName: \"kubernetes.io/projected/337d4867-fb09-4498-a2a9-72285e0e1a10-kube-api-access-hrc2x\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.211030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.210976    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-service-ca\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.211030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.211011    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-oauth-config\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.312526 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.312441    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-serving-cert\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.312526 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.312488    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-console-config\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.312724 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.312552    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-oauth-serving-cert\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.312724 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.312589    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hrc2x\" (UniqueName: \"kubernetes.io/projected/337d4867-fb09-4498-a2a9-72285e0e1a10-kube-api-access-hrc2x\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.312862 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.312718    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-service-ca\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.312862 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.312767    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-oauth-config\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.313297 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.313268    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-oauth-serving-cert\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.313386 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.313267    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-console-config\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.313386 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.313356    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-service-ca\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.315102 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.315081    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-serving-cert\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.315153 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.315103    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-oauth-config\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.322183 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.322158    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrc2x\" (UniqueName: \"kubernetes.io/projected/337d4867-fb09-4498-a2a9-72285e0e1a10-kube-api-access-hrc2x\") pod \"console-85c94b6569-bpvw8\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") " pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.357898 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.357861    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:26.489093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.489052    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c94b6569-bpvw8"]
May 20 13:53:26.493196 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:26.493168    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337d4867_fb09_4498_a2a9_72285e0e1a10.slice/crio-5a1a49ff82df87c942bb0773f0ba019a2df68a4536f4035f7eb5bd336da39cb1 WatchSource:0}: Error finding container 5a1a49ff82df87c942bb0773f0ba019a2df68a4536f4035f7eb5bd336da39cb1: Status 404 returned error can't find the container with id 5a1a49ff82df87c942bb0773f0ba019a2df68a4536f4035f7eb5bd336da39cb1
May 20 13:53:26.823731 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:26.823693    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c94b6569-bpvw8" event={"ID":"337d4867-fb09-4498-a2a9-72285e0e1a10","Type":"ContainerStarted","Data":"5a1a49ff82df87c942bb0773f0ba019a2df68a4536f4035f7eb5bd336da39cb1"}
May 20 13:53:27.628448 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.628412    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-76bdc4d5cf-xgvlk"]
May 20 13:53:27.630723 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.630692    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.640299 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.640241    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 20 13:53:27.646745 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.646716    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bdc4d5cf-xgvlk"]
May 20 13:53:27.726710 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.726654    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpkr\" (UniqueName: \"kubernetes.io/projected/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-kube-api-access-kbpkr\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.726935 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.726718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-serving-cert\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.726935 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.726795    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-oauth-config\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.726935 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.726890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-service-ca\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.726935 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.726917    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-oauth-serving-cert\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.727140 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.726955    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-trusted-ca-bundle\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.727140 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.726990    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-config\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.828329 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.828291    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpkr\" (UniqueName: \"kubernetes.io/projected/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-kube-api-access-kbpkr\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.828504 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.828341    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-serving-cert\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.828504 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.828410    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-oauth-config\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.828504 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.828457    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-service-ca\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.828504 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.828477    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-oauth-serving-cert\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.828712 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.828516    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-trusted-ca-bundle\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.828712 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.828549    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-config\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.829341 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.829292    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-config\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.829937 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.829611    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-service-ca\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.829937 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.829878    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-oauth-serving-cert\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.830214 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.830194    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-trusted-ca-bundle\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.831385 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.831353    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-serving-cert\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.831867 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.831815    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-oauth-config\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.836353 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.836332    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpkr\" (UniqueName: \"kubernetes.io/projected/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-kube-api-access-kbpkr\") pod \"console-76bdc4d5cf-xgvlk\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") " pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:27.943027 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:27.942975    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:28.083880 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:28.083828    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bdc4d5cf-xgvlk"]
May 20 13:53:29.038875 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.038827    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xp85s"]
May 20 13:53:29.047300 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.047266    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.049751 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.049726    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vz65g\""
May 20 13:53:29.050250 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.050227    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 13:53:29.050727 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.050707    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 13:53:29.051183 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.050718    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 13:53:29.051183 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.050990    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 13:53:29.051183 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.050824    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 13:53:29.051183 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.050730    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 13:53:29.064792 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:29.064751    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d92e8b2_b253_40ad_85f4_f35ffcf3ed17.slice/crio-05ec6786c1d010528a0d35e19c47c781aaf3a02648cf61faac64188cbc71edea WatchSource:0}: Error finding container 05ec6786c1d010528a0d35e19c47c781aaf3a02648cf61faac64188cbc71edea: Status 404 returned error can't find the container with id 05ec6786c1d010528a0d35e19c47c781aaf3a02648cf61faac64188cbc71edea
May 20 13:53:29.137848 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.137811    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138017 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.137875    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgfr\" (UniqueName: \"kubernetes.io/projected/558b53d4-03e9-4e11-a495-b63b06d7c27b-kube-api-access-tsgfr\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138017 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.137941    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-wtmp\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138017 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.137992    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-root\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138251 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.138028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558b53d4-03e9-4e11-a495-b63b06d7c27b-metrics-client-ca\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138251 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.138057    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138251 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.138092    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-textfile\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138251 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.138123    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-sys\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.138585 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.138278    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.247553 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247503    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-textfile\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.247553 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247556    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-sys\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.247802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247667    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.247802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247715    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.247802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247741    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgfr\" (UniqueName: \"kubernetes.io/projected/558b53d4-03e9-4e11-a495-b63b06d7c27b-kube-api-access-tsgfr\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.247802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247767    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-wtmp\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.247802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247790    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-root\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247823    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558b53d4-03e9-4e11-a495-b63b06d7c27b-metrics-client-ca\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.247942    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.248209    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-textfile\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.248247    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-wtmp\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.248257    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-sys\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.248313    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/558b53d4-03e9-4e11-a495-b63b06d7c27b-root\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:29.248318    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 13:53:29.248787 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:29.248374    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls podName:558b53d4-03e9-4e11-a495-b63b06d7c27b nodeName:}" failed. No retries permitted until 2026-05-20 13:53:29.748354619 +0000 UTC m=+194.371079669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls") pod "node-exporter-xp85s" (UID: "558b53d4-03e9-4e11-a495-b63b06d7c27b") : secret "node-exporter-tls" not found
May 20 13:53:29.249353 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.249327    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-accelerators-collector-config\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.250216 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.249672    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558b53d4-03e9-4e11-a495-b63b06d7c27b-metrics-client-ca\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.251514 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.251472    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.258490 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.258427    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgfr\" (UniqueName: \"kubernetes.io/projected/558b53d4-03e9-4e11-a495-b63b06d7c27b-kube-api-access-tsgfr\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.752715 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.752671    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:29.752883 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:29.752821    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 13:53:29.752953 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:29.752908    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls podName:558b53d4-03e9-4e11-a495-b63b06d7c27b nodeName:}" failed. No retries permitted until 2026-05-20 13:53:30.752892334 +0000 UTC m=+195.375617385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls") pod "node-exporter-xp85s" (UID: "558b53d4-03e9-4e11-a495-b63b06d7c27b") : secret "node-exporter-tls" not found
May 20 13:53:29.833433 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.833395    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c94b6569-bpvw8" event={"ID":"337d4867-fb09-4498-a2a9-72285e0e1a10","Type":"ContainerStarted","Data":"58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0"}
May 20 13:53:29.834883 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.834854    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bdc4d5cf-xgvlk" event={"ID":"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17","Type":"ContainerStarted","Data":"309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c"}
May 20 13:53:29.834982 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.834889    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bdc4d5cf-xgvlk" event={"ID":"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17","Type":"ContainerStarted","Data":"05ec6786c1d010528a0d35e19c47c781aaf3a02648cf61faac64188cbc71edea"}
May 20 13:53:29.851675 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.851608    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-85c94b6569-bpvw8" podStartSLOduration=1.245936947 podStartE2EDuration="3.851593092s" podCreationTimestamp="2026-05-20 13:53:26 +0000 UTC" firstStartedPulling="2026-05-20 13:53:26.495566604 +0000 UTC m=+191.118291651" lastFinishedPulling="2026-05-20 13:53:29.101222746 +0000 UTC m=+193.723947796" observedRunningTime="2026-05-20 13:53:29.850869189 +0000 UTC m=+194.473594263" watchObservedRunningTime="2026-05-20 13:53:29.851593092 +0000 UTC m=+194.474318160"
May 20 13:53:29.869737 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:29.869688    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-76bdc4d5cf-xgvlk" podStartSLOduration=2.590687039 podStartE2EDuration="2.869672968s" podCreationTimestamp="2026-05-20 13:53:27 +0000 UTC" firstStartedPulling="2026-05-20 13:53:29.067929316 +0000 UTC m=+193.690654369" lastFinishedPulling="2026-05-20 13:53:29.346915246 +0000 UTC m=+193.969640298" observedRunningTime="2026-05-20 13:53:29.869210708 +0000 UTC m=+194.491935775" watchObservedRunningTime="2026-05-20 13:53:29.869672968 +0000 UTC m=+194.492398036"
May 20 13:53:30.763867 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:30.763806    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:30.766156 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:30.766134    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b53d4-03e9-4e11-a495-b63b06d7c27b-node-exporter-tls\") pod \"node-exporter-xp85s\" (UID: \"558b53d4-03e9-4e11-a495-b63b06d7c27b\") " pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:30.858213 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:30.858174    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xp85s"
May 20 13:53:30.867987 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:30.867955    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558b53d4_03e9_4e11_a495_b63b06d7c27b.slice/crio-ed826acf39bfda97bcb26de66951e16aafc891efcfd8cd610376357080e94f4c WatchSource:0}: Error finding container ed826acf39bfda97bcb26de66951e16aafc891efcfd8cd610376357080e94f4c: Status 404 returned error can't find the container with id ed826acf39bfda97bcb26de66951e16aafc891efcfd8cd610376357080e94f4c
May 20 13:53:31.842646 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:31.842610    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp85s" event={"ID":"558b53d4-03e9-4e11-a495-b63b06d7c27b","Type":"ContainerStarted","Data":"9235d9323a19b1568ff7427a7e65792f9e9767d92c19bad5a200b6ddc6af87d4"}
May 20 13:53:31.842646 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:31.842654    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp85s" event={"ID":"558b53d4-03e9-4e11-a495-b63b06d7c27b","Type":"ContainerStarted","Data":"ed826acf39bfda97bcb26de66951e16aafc891efcfd8cd610376357080e94f4c"}
May 20 13:53:32.022995 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.022902    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dd5b6b687-2d944"]
May 20 13:53:32.025559 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.025535    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.030038 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.030014    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 20 13:53:32.030038 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.030018    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 20 13:53:32.030682 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.030655    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jgjzx\""
May 20 13:53:32.030802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.030715    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 20 13:53:32.031509 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.031490    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 20 13:53:32.031660 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.031638    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-17gfmpvd4g7hn\""
May 20 13:53:32.031660 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.031646    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 20 13:53:32.045029 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.045002    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dd5b6b687-2d944"]
May 20 13:53:32.077619 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.077766 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077641    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.077766 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-tls\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.077887 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-grpc-tls\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.077887 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077832    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.077970 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077887    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.077970 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077917    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-metrics-client-ca\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.077970 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.077946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flftn\" (UniqueName: \"kubernetes.io/projected/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-kube-api-access-flftn\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.178811 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.178779    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-tls\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.179005 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.178822    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-grpc-tls\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.179005 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.178867    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.179005 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.178919    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.179157 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.179074    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-metrics-client-ca\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.179157 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.179116    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-flftn\" (UniqueName: \"kubernetes.io/projected/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-kube-api-access-flftn\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.179254 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.179201    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.179305 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.179286    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.180032 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.180007    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-metrics-client-ca\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.182103 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.182029    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.182241 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.182218    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-tls\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.182300 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.182246    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.182300 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.182285    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-grpc-tls\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.182376 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.182319    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.182471 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.182453    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.190466 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.190441    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flftn\" (UniqueName: \"kubernetes.io/projected/8636cff6-00f2-452a-b5aa-5a2c19fd5df9-kube-api-access-flftn\") pod \"thanos-querier-7dd5b6b687-2d944\" (UID: \"8636cff6-00f2-452a-b5aa-5a2c19fd5df9\") " pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.335002 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.334927    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:32.469596 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.469563    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dd5b6b687-2d944"]
May 20 13:53:32.472358 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:32.472328    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8636cff6_00f2_452a_b5aa_5a2c19fd5df9.slice/crio-1bdbf8b733aa6ab1361cb8d03d4c36371b443bd2a0d7584ce740f412c65edb70 WatchSource:0}: Error finding container 1bdbf8b733aa6ab1361cb8d03d4c36371b443bd2a0d7584ce740f412c65edb70: Status 404 returned error can't find the container with id 1bdbf8b733aa6ab1361cb8d03d4c36371b443bd2a0d7584ce740f412c65edb70
May 20 13:53:32.847251 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.847217    2554 generic.go:358] "Generic (PLEG): container finished" podID="558b53d4-03e9-4e11-a495-b63b06d7c27b" containerID="9235d9323a19b1568ff7427a7e65792f9e9767d92c19bad5a200b6ddc6af87d4" exitCode=0
May 20 13:53:32.847674 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.847295    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp85s" event={"ID":"558b53d4-03e9-4e11-a495-b63b06d7c27b","Type":"ContainerDied","Data":"9235d9323a19b1568ff7427a7e65792f9e9767d92c19bad5a200b6ddc6af87d4"}
May 20 13:53:32.848332 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:32.848312    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" event={"ID":"8636cff6-00f2-452a-b5aa-5a2c19fd5df9","Type":"ContainerStarted","Data":"1bdbf8b733aa6ab1361cb8d03d4c36371b443bd2a0d7584ce740f412c65edb70"}
May 20 13:53:33.780346 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.780307    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"]
May 20 13:53:33.783010 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.782983    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:33.785508 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.785486    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 20 13:53:33.785622 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.785517    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-4rrll\""
May 20 13:53:33.794819 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.794793    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"]
May 20 13:53:33.798582 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.798365    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1d0459a-6c74-4baa-80eb-53e8e1039f03-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-9w95s\" (UID: \"c1d0459a-6c74-4baa-80eb-53e8e1039f03\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:33.853624 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.853579    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp85s" event={"ID":"558b53d4-03e9-4e11-a495-b63b06d7c27b","Type":"ContainerStarted","Data":"13f9b780b4352ff7905cf3a59dd657199ecd35ebc0c8b7ef450e497df9b5ce89"}
May 20 13:53:33.854102 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.853638    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp85s" event={"ID":"558b53d4-03e9-4e11-a495-b63b06d7c27b","Type":"ContainerStarted","Data":"9e452eac72bd8d6f8644c334708191c2e1e233766234fc2d76229a586471e7f9"}
May 20 13:53:33.867426 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.867390    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c94b6569-bpvw8"]
May 20 13:53:33.877042 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.876992    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xp85s" podStartSLOduration=4.011261028 podStartE2EDuration="4.876973654s" podCreationTimestamp="2026-05-20 13:53:29 +0000 UTC" firstStartedPulling="2026-05-20 13:53:30.869877876 +0000 UTC m=+195.492602933" lastFinishedPulling="2026-05-20 13:53:31.735590514 +0000 UTC m=+196.358315559" observedRunningTime="2026-05-20 13:53:33.874736426 +0000 UTC m=+198.497461494" watchObservedRunningTime="2026-05-20 13:53:33.876973654 +0000 UTC m=+198.499698723"
May 20 13:53:33.899725 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.899687    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1d0459a-6c74-4baa-80eb-53e8e1039f03-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-9w95s\" (UID: \"c1d0459a-6c74-4baa-80eb-53e8e1039f03\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:33.899902 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:33.899874    2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 20 13:53:33.899989 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:33.899952    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1d0459a-6c74-4baa-80eb-53e8e1039f03-monitoring-plugin-cert podName:c1d0459a-6c74-4baa-80eb-53e8e1039f03 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:34.399929362 +0000 UTC m=+199.022654412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c1d0459a-6c74-4baa-80eb-53e8e1039f03-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-9w95s" (UID: "c1d0459a-6c74-4baa-80eb-53e8e1039f03") : secret "monitoring-plugin-cert" not found
May 20 13:53:33.907549 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.907518    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-585fdf4544-ltdpt"]
May 20 13:53:33.909998 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.909970    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:33.930193 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:33.930152    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585fdf4544-ltdpt"]
May 20 13:53:34.000803 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.000764    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7dx\" (UniqueName: \"kubernetes.io/projected/8a678977-b24d-402a-8612-c2490f78dc6b-kube-api-access-ph7dx\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.000989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.000891    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-trusted-ca-bundle\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.000989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.000914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-oauth-serving-cert\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.000989 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.000940    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-serving-cert\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.001172 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.001079    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-oauth-config\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.001172 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.001123    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-console-config\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.001247 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.001203    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-service-ca\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.101992 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.101902    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-serving-cert\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.102177 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.102017    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-oauth-config\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.102177 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.102044    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-console-config\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.102177 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.102155    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-service-ca\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.102318 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.102239    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7dx\" (UniqueName: \"kubernetes.io/projected/8a678977-b24d-402a-8612-c2490f78dc6b-kube-api-access-ph7dx\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.102422 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.102329    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-trusted-ca-bundle\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.102422 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.102358    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-oauth-serving-cert\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.102824 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.102800    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-console-config\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.103052 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.103027    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-service-ca\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.103151 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.103111    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-oauth-serving-cert\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.103465 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.103439    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-trusted-ca-bundle\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.105148 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.105124    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-serving-cert\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.105238 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.105178    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-oauth-config\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.111537 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.111513    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7dx\" (UniqueName: \"kubernetes.io/projected/8a678977-b24d-402a-8612-c2490f78dc6b-kube-api-access-ph7dx\") pod \"console-585fdf4544-ltdpt\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") " pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.221287 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.221263    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:34.361613 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.361531    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585fdf4544-ltdpt"]
May 20 13:53:34.365276 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:34.365244    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a678977_b24d_402a_8612_c2490f78dc6b.slice/crio-adf830439ce60f20f41046b4b73ce443b76ce85142cfa0ee0ca9e30481af437b WatchSource:0}: Error finding container adf830439ce60f20f41046b4b73ce443b76ce85142cfa0ee0ca9e30481af437b: Status 404 returned error can't find the container with id adf830439ce60f20f41046b4b73ce443b76ce85142cfa0ee0ca9e30481af437b
May 20 13:53:34.406159 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.406131    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1d0459a-6c74-4baa-80eb-53e8e1039f03-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-9w95s\" (UID: \"c1d0459a-6c74-4baa-80eb-53e8e1039f03\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:34.408780 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.408755    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1d0459a-6c74-4baa-80eb-53e8e1039f03-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-9w95s\" (UID: \"c1d0459a-6c74-4baa-80eb-53e8e1039f03\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:34.695414 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.695378    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:34.826291 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.826234    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"]
May 20 13:53:34.829916 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:53:34.829881    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d0459a_6c74_4baa_80eb_53e8e1039f03.slice/crio-1bdfb4b4e0f7d6a40dc26101446a3fbc9b02bbb146edfd8ed30a4103f15efd61 WatchSource:0}: Error finding container 1bdfb4b4e0f7d6a40dc26101446a3fbc9b02bbb146edfd8ed30a4103f15efd61: Status 404 returned error can't find the container with id 1bdfb4b4e0f7d6a40dc26101446a3fbc9b02bbb146edfd8ed30a4103f15efd61
May 20 13:53:34.859585 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.859545    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" event={"ID":"8636cff6-00f2-452a-b5aa-5a2c19fd5df9","Type":"ContainerStarted","Data":"906dc039523724296d2ad242ba8b437d379877b1d221f3615ce8428712b3a457"}
May 20 13:53:34.860078 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.859593    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" event={"ID":"8636cff6-00f2-452a-b5aa-5a2c19fd5df9","Type":"ContainerStarted","Data":"67f5391ee8b673a116ee54f23a6fdd95198bc551f2d99755ec2dff8578cd3380"}
May 20 13:53:34.860078 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.859605    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" event={"ID":"8636cff6-00f2-452a-b5aa-5a2c19fd5df9","Type":"ContainerStarted","Data":"0f2d5912a9069b3a12bb0b7b8ef7741dc37c079ffbf24ce8335bcf24d314b4a7"}
May 20 13:53:34.861280 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.861256    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fdf4544-ltdpt" event={"ID":"8a678977-b24d-402a-8612-c2490f78dc6b","Type":"ContainerStarted","Data":"4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd"}
May 20 13:53:34.861405 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.861287    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fdf4544-ltdpt" event={"ID":"8a678977-b24d-402a-8612-c2490f78dc6b","Type":"ContainerStarted","Data":"adf830439ce60f20f41046b4b73ce443b76ce85142cfa0ee0ca9e30481af437b"}
May 20 13:53:34.862567 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.862540    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s" event={"ID":"c1d0459a-6c74-4baa-80eb-53e8e1039f03","Type":"ContainerStarted","Data":"1bdfb4b4e0f7d6a40dc26101446a3fbc9b02bbb146edfd8ed30a4103f15efd61"}
May 20 13:53:34.886854 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:34.886792    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-585fdf4544-ltdpt" podStartSLOduration=1.8867729149999999 podStartE2EDuration="1.886772915s" podCreationTimestamp="2026-05-20 13:53:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:53:34.884742706 +0000 UTC m=+199.507467775" watchObservedRunningTime="2026-05-20 13:53:34.886772915 +0000 UTC m=+199.509497984"
May 20 13:53:35.868524 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:35.868480    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" event={"ID":"8636cff6-00f2-452a-b5aa-5a2c19fd5df9","Type":"ContainerStarted","Data":"7fa7cd9166b5f451751bbf43a74db0ac492482bd4ba968f678a671eebcfacdf5"}
May 20 13:53:35.868982 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:35.868531    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" event={"ID":"8636cff6-00f2-452a-b5aa-5a2c19fd5df9","Type":"ContainerStarted","Data":"6d4e6b31f7201a7e5be794b7c00c083677b37e6e3dbf0ed9140a8bba71dc5b6f"}
May 20 13:53:35.868982 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:35.868545    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" event={"ID":"8636cff6-00f2-452a-b5aa-5a2c19fd5df9","Type":"ContainerStarted","Data":"8c80a42605f5b4c9389dc9db236678c75e20d86cb7f0c611107fc1ef0e00556b"}
May 20 13:53:35.895404 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:35.895349    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944" podStartSLOduration=1.322776129 podStartE2EDuration="3.895334296s" podCreationTimestamp="2026-05-20 13:53:32 +0000 UTC" firstStartedPulling="2026-05-20 13:53:32.474122492 +0000 UTC m=+197.096847542" lastFinishedPulling="2026-05-20 13:53:35.046680664 +0000 UTC m=+199.669405709" observedRunningTime="2026-05-20 13:53:35.89358151 +0000 UTC m=+200.516306611" watchObservedRunningTime="2026-05-20 13:53:35.895334296 +0000 UTC m=+200.518059361"
May 20 13:53:36.358159 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:36.358104    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:36.872820 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:36.872770    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s" event={"ID":"c1d0459a-6c74-4baa-80eb-53e8e1039f03","Type":"ContainerStarted","Data":"d771bb524f45b1254df98214c4a7c3a295a4ea8a583568db47be2544bd4e0d61"}
May 20 13:53:36.873293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:36.873067    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:36.873293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:36.873131    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:36.877661 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:36.877633    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s"
May 20 13:53:36.888924 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:36.888878    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-9w95s" podStartSLOduration=2.593605665 podStartE2EDuration="3.888865753s" podCreationTimestamp="2026-05-20 13:53:33 +0000 UTC" firstStartedPulling="2026-05-20 13:53:34.832036142 +0000 UTC m=+199.454761189" lastFinishedPulling="2026-05-20 13:53:36.127296227 +0000 UTC m=+200.750021277" observedRunningTime="2026-05-20 13:53:36.888088186 +0000 UTC m=+201.510813252" watchObservedRunningTime="2026-05-20 13:53:36.888865753 +0000 UTC m=+201.511590818"
May 20 13:53:37.943658 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:37.943629    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:37.943658 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:37.943669    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:37.948209 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:37.948187    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:38.884357 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:38.884328    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:53:39.436528 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.436490    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"]
May 20 13:53:39.436922 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:39.436729    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt" podUID="da6f1cd9-dbc0-445c-a6d4-dc3497d10893"
May 20 13:53:39.883080 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.882995    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:53:39.887596 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.887569    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:53:39.958078 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958043    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-trusted-ca\") pod \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") "
May 20 13:53:39.958078 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958081    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jffbf\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-kube-api-access-jffbf\") pod \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") "
May 20 13:53:39.958293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958110    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-bound-sa-token\") pod \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") "
May 20 13:53:39.958293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958143    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-ca-trust-extracted\") pod \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") "
May 20 13:53:39.958293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958166    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-installation-pull-secrets\") pod \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") "
May 20 13:53:39.958293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958205    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-image-registry-private-configuration\") pod \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") "
May 20 13:53:39.958490 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958341    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-certificates\") pod \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\" (UID: \"da6f1cd9-dbc0-445c-a6d4-dc3497d10893\") "
May 20 13:53:39.958490 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958461    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "da6f1cd9-dbc0-445c-a6d4-dc3497d10893" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:53:39.958587 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958487    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "da6f1cd9-dbc0-445c-a6d4-dc3497d10893" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:39.958759 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.958734    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "da6f1cd9-dbc0-445c-a6d4-dc3497d10893" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:39.959107 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.959075    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-certificates\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:39.959209 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.959117    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-trusted-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:39.959209 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.959137    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-ca-trust-extracted\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:39.960722 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.960695    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "da6f1cd9-dbc0-445c-a6d4-dc3497d10893" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:39.960828 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.960795    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "da6f1cd9-dbc0-445c-a6d4-dc3497d10893" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:53:39.960912 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.960816    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "da6f1cd9-dbc0-445c-a6d4-dc3497d10893" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:39.960912 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:39.960900    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-kube-api-access-jffbf" (OuterVolumeSpecName: "kube-api-access-jffbf") pod "da6f1cd9-dbc0-445c-a6d4-dc3497d10893" (UID: "da6f1cd9-dbc0-445c-a6d4-dc3497d10893"). InnerVolumeSpecName "kube-api-access-jffbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:53:40.060568 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.060525    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-image-registry-private-configuration\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:40.060568 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.060560    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jffbf\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-kube-api-access-jffbf\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:40.060568 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.060570    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-bound-sa-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:40.060568 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.060579    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-installation-pull-secrets\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:40.885995 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.885959    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"
May 20 13:53:40.917917 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.917884    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"]
May 20 13:53:40.919813 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.919784    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dfc8fbf59-4zfzt"]
May 20 13:53:40.969117 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:40.969082    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da6f1cd9-dbc0-445c-a6d4-dc3497d10893-registry-tls\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:42.175652 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:42.175617    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6f1cd9-dbc0-445c-a6d4-dc3497d10893" path="/var/lib/kubelet/pods/da6f1cd9-dbc0-445c-a6d4-dc3497d10893/volumes"
May 20 13:53:42.884343 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:42.884314    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dd5b6b687-2d944"
May 20 13:53:44.221854 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:44.221817    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:44.222226 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:44.221888    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:44.226485 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:44.226462    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:44.901118 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:44.901081    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:53:44.950947 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:44.950910    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bdc4d5cf-xgvlk"]
May 20 13:53:58.894934 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:58.894861    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-85c94b6569-bpvw8" podUID="337d4867-fb09-4498-a2a9-72285e0e1a10" containerName="console" containerID="cri-o://58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0" gracePeriod=15
May 20 13:53:59.141021 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.140993    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85c94b6569-bpvw8_337d4867-fb09-4498-a2a9-72285e0e1a10/console/0.log"
May 20 13:53:59.141148 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.141052    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:59.233108 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233070    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-serving-cert\") pod \"337d4867-fb09-4498-a2a9-72285e0e1a10\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") "
May 20 13:53:59.233108 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233115    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrc2x\" (UniqueName: \"kubernetes.io/projected/337d4867-fb09-4498-a2a9-72285e0e1a10-kube-api-access-hrc2x\") pod \"337d4867-fb09-4498-a2a9-72285e0e1a10\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") "
May 20 13:53:59.233334 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233151    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-console-config\") pod \"337d4867-fb09-4498-a2a9-72285e0e1a10\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") "
May 20 13:53:59.233334 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233193    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-service-ca\") pod \"337d4867-fb09-4498-a2a9-72285e0e1a10\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") "
May 20 13:53:59.233334 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233251    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-oauth-serving-cert\") pod \"337d4867-fb09-4498-a2a9-72285e0e1a10\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") "
May 20 13:53:59.233334 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233276    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-oauth-config\") pod \"337d4867-fb09-4498-a2a9-72285e0e1a10\" (UID: \"337d4867-fb09-4498-a2a9-72285e0e1a10\") "
May 20 13:53:59.233711 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233672    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "337d4867-fb09-4498-a2a9-72285e0e1a10" (UID: "337d4867-fb09-4498-a2a9-72285e0e1a10"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:59.233711 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233684    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-service-ca" (OuterVolumeSpecName: "service-ca") pod "337d4867-fb09-4498-a2a9-72285e0e1a10" (UID: "337d4867-fb09-4498-a2a9-72285e0e1a10"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:59.233711 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.233667    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-console-config" (OuterVolumeSpecName: "console-config") pod "337d4867-fb09-4498-a2a9-72285e0e1a10" (UID: "337d4867-fb09-4498-a2a9-72285e0e1a10"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:59.235492 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.235470    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "337d4867-fb09-4498-a2a9-72285e0e1a10" (UID: "337d4867-fb09-4498-a2a9-72285e0e1a10"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:59.235578 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.235497    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "337d4867-fb09-4498-a2a9-72285e0e1a10" (UID: "337d4867-fb09-4498-a2a9-72285e0e1a10"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:59.235578 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.235557    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337d4867-fb09-4498-a2a9-72285e0e1a10-kube-api-access-hrc2x" (OuterVolumeSpecName: "kube-api-access-hrc2x") pod "337d4867-fb09-4498-a2a9-72285e0e1a10" (UID: "337d4867-fb09-4498-a2a9-72285e0e1a10"). InnerVolumeSpecName "kube-api-access-hrc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:53:59.334551 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.334504    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:59.334551 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.334546    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-oauth-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:59.334809 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.334562    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-oauth-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:59.334809 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.334579    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/337d4867-fb09-4498-a2a9-72285e0e1a10-console-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:59.334809 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.334594    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrc2x\" (UniqueName: \"kubernetes.io/projected/337d4867-fb09-4498-a2a9-72285e0e1a10-kube-api-access-hrc2x\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:59.334809 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.334610    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/337d4867-fb09-4498-a2a9-72285e0e1a10-console-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:53:59.936271 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.936238    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85c94b6569-bpvw8_337d4867-fb09-4498-a2a9-72285e0e1a10/console/0.log"
May 20 13:53:59.936718 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.936280    2554 generic.go:358] "Generic (PLEG): container finished" podID="337d4867-fb09-4498-a2a9-72285e0e1a10" containerID="58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0" exitCode=2
May 20 13:53:59.936718 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.936342    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c94b6569-bpvw8"
May 20 13:53:59.936718 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.936346    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c94b6569-bpvw8" event={"ID":"337d4867-fb09-4498-a2a9-72285e0e1a10","Type":"ContainerDied","Data":"58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0"}
May 20 13:53:59.936718 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.936382    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c94b6569-bpvw8" event={"ID":"337d4867-fb09-4498-a2a9-72285e0e1a10","Type":"ContainerDied","Data":"5a1a49ff82df87c942bb0773f0ba019a2df68a4536f4035f7eb5bd336da39cb1"}
May 20 13:53:59.936718 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.936405    2554 scope.go:117] "RemoveContainer" containerID="58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0"
May 20 13:53:59.945630 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.945528    2554 scope.go:117] "RemoveContainer" containerID="58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0"
May 20 13:53:59.946037 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:53:59.945939    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0\": container with ID starting with 58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0 not found: ID does not exist" containerID="58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0"
May 20 13:53:59.946037 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.945977    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0"} err="failed to get container status \"58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0\": rpc error: code = NotFound desc = could not find container \"58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0\": container with ID starting with 58caf84a54e2cea75a1cd542870f48aca9625cce24acd12f875fdbb8a1e660a0 not found: ID does not exist"
May 20 13:53:59.959042 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.958996    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c94b6569-bpvw8"]
May 20 13:53:59.962832 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:53:59.962802    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c94b6569-bpvw8"]
May 20 13:54:00.177004 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:00.176970    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="337d4867-fb09-4498-a2a9-72285e0e1a10" path="/var/lib/kubelet/pods/337d4867-fb09-4498-a2a9-72285e0e1a10/volumes"
May 20 13:54:05.956203 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:05.956169    2554 generic.go:358] "Generic (PLEG): container finished" podID="61e2746b-3239-4211-8013-f40b5b0f7ccc" containerID="030fb2707063845779da55e522e9a33abda066f0ff863e9192100a30928c1379" exitCode=0
May 20 13:54:05.956722 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:05.956251    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8mb69" event={"ID":"61e2746b-3239-4211-8013-f40b5b0f7ccc","Type":"ContainerDied","Data":"030fb2707063845779da55e522e9a33abda066f0ff863e9192100a30928c1379"}
May 20 13:54:05.956722 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:05.956668    2554 scope.go:117] "RemoveContainer" containerID="030fb2707063845779da55e522e9a33abda066f0ff863e9192100a30928c1379"
May 20 13:54:05.957560 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:05.957538    2554 generic.go:358] "Generic (PLEG): container finished" podID="9a06524c-de8f-4f26-8ba4-ae198f93fdbf" containerID="4ae8bad1ee6110f97dc03d154ba7a82d9259cf8edf43c5b1d750a60be11887b7" exitCode=0
May 20 13:54:05.957630 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:05.957578    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2" event={"ID":"9a06524c-de8f-4f26-8ba4-ae198f93fdbf","Type":"ContainerDied","Data":"4ae8bad1ee6110f97dc03d154ba7a82d9259cf8edf43c5b1d750a60be11887b7"}
May 20 13:54:05.957910 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:05.957894    2554 scope.go:117] "RemoveContainer" containerID="4ae8bad1ee6110f97dc03d154ba7a82d9259cf8edf43c5b1d750a60be11887b7"
May 20 13:54:06.961908 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:06.961872    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-8mb69" event={"ID":"61e2746b-3239-4211-8013-f40b5b0f7ccc","Type":"ContainerStarted","Data":"87b4b47064a5c503525a1f961644a98e606c391b1b1d267724cc12410dfcf6f8"}
May 20 13:54:06.963390 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:06.963365    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-mcdf2" event={"ID":"9a06524c-de8f-4f26-8ba4-ae198f93fdbf","Type":"ContainerStarted","Data":"032e97e431a852c1c786888504dc27f0c11e71d92097e5210c3c34a3bea2b3db"}
May 20 13:54:09.970750 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:09.970676    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-76bdc4d5cf-xgvlk" podUID="2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" containerName="console" containerID="cri-o://309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c" gracePeriod=15
May 20 13:54:10.213939 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.213914    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76bdc4d5cf-xgvlk_2d92e8b2-b253-40ad-85f4-f35ffcf3ed17/console/0.log"
May 20 13:54:10.214065 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.213976    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:54:10.226872 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.226802    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbpkr\" (UniqueName: \"kubernetes.io/projected/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-kube-api-access-kbpkr\") pod \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") "
May 20 13:54:10.226872 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.226854    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-oauth-serving-cert\") pod \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") "
May 20 13:54:10.227007 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.226891    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-serving-cert\") pod \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") "
May 20 13:54:10.227007 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.226910    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-config\") pod \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") "
May 20 13:54:10.227007 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.226927    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-oauth-config\") pod \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") "
May 20 13:54:10.227007 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.226969    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-service-ca\") pod \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") "
May 20 13:54:10.227007 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.227000    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-trusted-ca-bundle\") pod \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\" (UID: \"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17\") "
May 20 13:54:10.227407 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.227379    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-config" (OuterVolumeSpecName: "console-config") pod "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" (UID: "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:10.227489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.227414    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" (UID: "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:10.227547 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.227511    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-service-ca" (OuterVolumeSpecName: "service-ca") pod "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" (UID: "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:10.227603 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.227546    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" (UID: "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:10.229031 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.229012    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" (UID: "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:10.229133 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.229041    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-kube-api-access-kbpkr" (OuterVolumeSpecName: "kube-api-access-kbpkr") pod "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" (UID: "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17"). InnerVolumeSpecName "kube-api-access-kbpkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:10.229133 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.229069    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" (UID: "2d92e8b2-b253-40ad-85f4-f35ffcf3ed17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:10.328195 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.328163    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:54:10.328195 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.328190    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-trusted-ca-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:54:10.328195 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.328200    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpkr\" (UniqueName: \"kubernetes.io/projected/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-kube-api-access-kbpkr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:54:10.328422 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.328209    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-oauth-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:54:10.328422 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.328220    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:54:10.328422 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.328229    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:54:10.328422 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.328238    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17-console-oauth-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:54:10.975093 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.975060    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76bdc4d5cf-xgvlk_2d92e8b2-b253-40ad-85f4-f35ffcf3ed17/console/0.log"
May 20 13:54:10.975507 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.975108    2554 generic.go:358] "Generic (PLEG): container finished" podID="2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" containerID="309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c" exitCode=2
May 20 13:54:10.975507 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.975167    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bdc4d5cf-xgvlk"
May 20 13:54:10.975507 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.975185    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bdc4d5cf-xgvlk" event={"ID":"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17","Type":"ContainerDied","Data":"309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c"}
May 20 13:54:10.975507 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.975216    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bdc4d5cf-xgvlk" event={"ID":"2d92e8b2-b253-40ad-85f4-f35ffcf3ed17","Type":"ContainerDied","Data":"05ec6786c1d010528a0d35e19c47c781aaf3a02648cf61faac64188cbc71edea"}
May 20 13:54:10.975507 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.975230    2554 scope.go:117] "RemoveContainer" containerID="309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c"
May 20 13:54:10.983417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.983395    2554 scope.go:117] "RemoveContainer" containerID="309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c"
May 20 13:54:10.983700 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:54:10.983682    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c\": container with ID starting with 309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c not found: ID does not exist" containerID="309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c"
May 20 13:54:10.983753 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.983709    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c"} err="failed to get container status \"309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c\": rpc error: code = NotFound desc = could not find container \"309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c\": container with ID starting with 309f8d1ba21f5d1f7195c67419c0c47def56cdeccc874331e0e1752c6136a43c not found: ID does not exist"
May 20 13:54:10.997666 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:10.997637    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bdc4d5cf-xgvlk"]
May 20 13:54:11.001374 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:11.001354    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76bdc4d5cf-xgvlk"]
May 20 13:54:12.176250 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:12.176213    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" path="/var/lib/kubelet/pods/2d92e8b2-b253-40ad-85f4-f35ffcf3ed17/volumes"
May 20 13:54:22.006752 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:22.006709    2554 generic.go:358] "Generic (PLEG): container finished" podID="401a135b-3831-4cd9-944c-b230c3a84a91" containerID="f801d55d3f6d3c600e22015facef0cd4b10d70651d2a338a8440b0e635537f29" exitCode=0
May 20 13:54:22.007188 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:22.006769    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r" event={"ID":"401a135b-3831-4cd9-944c-b230c3a84a91","Type":"ContainerDied","Data":"f801d55d3f6d3c600e22015facef0cd4b10d70651d2a338a8440b0e635537f29"}
May 20 13:54:22.007188 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:22.007071    2554 scope.go:117] "RemoveContainer" containerID="f801d55d3f6d3c600e22015facef0cd4b10d70651d2a338a8440b0e635537f29"
May 20 13:54:23.016524 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:23.016489    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-9d68r" event={"ID":"401a135b-3831-4cd9-944c-b230c3a84a91","Type":"ContainerStarted","Data":"82bb91a208b61ada126050b29f2bfe42cedd9ddc27e39d0509857e91e8dd7a62"}
May 20 13:54:27.881549 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:27.881508    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:54:27.884021 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:27.883993    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8c29338-36d1-4d0d-942e-6b5312426efc-metrics-certs\") pod \"network-metrics-daemon-n4cgv\" (UID: \"b8c29338-36d1-4d0d-942e-6b5312426efc\") " pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:54:27.976762 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:27.976728    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwf8m\""
May 20 13:54:27.984277 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:27.984252    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n4cgv"
May 20 13:54:28.117642 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:28.117592    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n4cgv"]
May 20 13:54:28.122089 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:54:28.122048    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c29338_36d1_4d0d_942e_6b5312426efc.slice/crio-a7abd1166baf95fab5577da16860ac04f28e2adb0cde81d3b8f1e1437690110d WatchSource:0}: Error finding container a7abd1166baf95fab5577da16860ac04f28e2adb0cde81d3b8f1e1437690110d: Status 404 returned error can't find the container with id a7abd1166baf95fab5577da16860ac04f28e2adb0cde81d3b8f1e1437690110d
May 20 13:54:29.037671 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:29.037621    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4cgv" event={"ID":"b8c29338-36d1-4d0d-942e-6b5312426efc","Type":"ContainerStarted","Data":"a7abd1166baf95fab5577da16860ac04f28e2adb0cde81d3b8f1e1437690110d"}
May 20 13:54:30.042079 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:30.042042    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4cgv" event={"ID":"b8c29338-36d1-4d0d-942e-6b5312426efc","Type":"ContainerStarted","Data":"3717fc602699934668ca0bf821247aae976436036d50bc5964c1100f46a7f2d5"}
May 20 13:54:30.042079 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:30.042082    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n4cgv" event={"ID":"b8c29338-36d1-4d0d-942e-6b5312426efc","Type":"ContainerStarted","Data":"a0c017ccbec20f6550c3b84f4aedef6ebfeb55a9cb20a98a2ca3eabd12ddaa2c"}
May 20 13:54:30.061532 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:30.061474    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n4cgv" podStartSLOduration=253.064412912 podStartE2EDuration="4m14.061458186s" podCreationTimestamp="2026-05-20 13:50:16 +0000 UTC" firstStartedPulling="2026-05-20 13:54:28.123895051 +0000 UTC m=+252.746620098" lastFinishedPulling="2026-05-20 13:54:29.120940312 +0000 UTC m=+253.743665372" observedRunningTime="2026-05-20 13:54:30.059093071 +0000 UTC m=+254.681818139" watchObservedRunningTime="2026-05-20 13:54:30.061458186 +0000 UTC m=+254.684183253"
May 20 13:54:56.720781 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:54:56.720733    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mtp2c" podUID="17a72c68-134e-4e0e-a8e7-e278d52ef7dc"
May 20 13:54:56.720781 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:54:56.720733    2554 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-gngpj" podUID="22742942-2c3a-4d4b-b354-8cf1476abc4e"
May 20 13:54:57.088313 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.088226    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cd74dd467-l9hsc"]
May 20 13:54:57.088666 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.088642    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" containerName="console"
May 20 13:54:57.088761 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.088669    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" containerName="console"
May 20 13:54:57.088761 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.088694    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337d4867-fb09-4498-a2a9-72285e0e1a10" containerName="console"
May 20 13:54:57.088761 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.088703    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="337d4867-fb09-4498-a2a9-72285e0e1a10" containerName="console"
May 20 13:54:57.088945 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.088775    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="337d4867-fb09-4498-a2a9-72285e0e1a10" containerName="console"
May 20 13:54:57.088945 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.088791    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d92e8b2-b253-40ad-85f4-f35ffcf3ed17" containerName="console"
May 20 13:54:57.093823 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.093798    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.105751 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.105717    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd74dd467-l9hsc"]
May 20 13:54:57.118311 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.118277    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:54:57.118467 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.118284    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtp2c"
May 20 13:54:57.143414 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.143377    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-service-ca\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.143590 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.143437    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-oauth-config\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.143590 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.143501    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-oauth-serving-cert\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.143590 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.143525    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-trusted-ca-bundle\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.143590 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.143543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb6n8\" (UniqueName: \"kubernetes.io/projected/eecdb085-eea1-4341-a932-ea1eab1921df-kube-api-access-xb6n8\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.143795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.143585    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-console-config\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.143795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.143636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-serving-cert\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.244613 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.244572    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-service-ca\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.244826 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.244652    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-oauth-config\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.244826 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.244686    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-oauth-serving-cert\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.244826 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.244720    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-trusted-ca-bundle\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.244826 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.244743    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xb6n8\" (UniqueName: \"kubernetes.io/projected/eecdb085-eea1-4341-a932-ea1eab1921df-kube-api-access-xb6n8\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.244826 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.244769    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-console-config\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.245116 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.244877    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-serving-cert\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.245635 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.245610    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-oauth-serving-cert\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.245828 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.245805    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-console-config\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.245982 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.245955    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-service-ca\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.246138 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.246004    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-trusted-ca-bundle\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.247424 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.247402    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-serving-cert\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.247548 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.247530    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-oauth-config\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.264223 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.264187    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb6n8\" (UniqueName: \"kubernetes.io/projected/eecdb085-eea1-4341-a932-ea1eab1921df-kube-api-access-xb6n8\") pod \"console-7cd74dd467-l9hsc\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") " pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.403265 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.403169    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:54:57.534609 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:57.534493    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd74dd467-l9hsc"]
May 20 13:54:57.537345 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:54:57.537316    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecdb085_eea1_4341_a932_ea1eab1921df.slice/crio-990794310bb548cc4623f2aa45ec8c05235cbe025c6e7fa0468a599b544f6301 WatchSource:0}: Error finding container 990794310bb548cc4623f2aa45ec8c05235cbe025c6e7fa0468a599b544f6301: Status 404 returned error can't find the container with id 990794310bb548cc4623f2aa45ec8c05235cbe025c6e7fa0468a599b544f6301
May 20 13:54:58.122218 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:58.122179    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd74dd467-l9hsc" event={"ID":"eecdb085-eea1-4341-a932-ea1eab1921df","Type":"ContainerStarted","Data":"b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616"}
May 20 13:54:58.122596 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:58.122225    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd74dd467-l9hsc" event={"ID":"eecdb085-eea1-4341-a932-ea1eab1921df","Type":"ContainerStarted","Data":"990794310bb548cc4623f2aa45ec8c05235cbe025c6e7fa0468a599b544f6301"}
May 20 13:54:58.141268 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:54:58.141225    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7cd74dd467-l9hsc" podStartSLOduration=1.141211173 podStartE2EDuration="1.141211173s" podCreationTimestamp="2026-05-20 13:54:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:54:58.139930383 +0000 UTC m=+282.762655451" watchObservedRunningTime="2026-05-20 13:54:58.141211173 +0000 UTC m=+282.763936240"
May 20 13:55:00.374428 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.374385    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:55:00.374961 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.374452    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:55:00.376749 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.376723    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17a72c68-134e-4e0e-a8e7-e278d52ef7dc-metrics-tls\") pod \"dns-default-mtp2c\" (UID: \"17a72c68-134e-4e0e-a8e7-e278d52ef7dc\") " pod="openshift-dns/dns-default-mtp2c"
May 20 13:55:00.376896 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.376875    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22742942-2c3a-4d4b-b354-8cf1476abc4e-cert\") pod \"ingress-canary-gngpj\" (UID: \"22742942-2c3a-4d4b-b354-8cf1476abc4e\") " pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:55:00.421510 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.421470    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxwgr\""
May 20 13:55:00.422197 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.422179    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxfgl\""
May 20 13:55:00.429183 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.429160    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gngpj"
May 20 13:55:00.429293 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.429176    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtp2c"
May 20 13:55:00.588068 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.588042    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gngpj"]
May 20 13:55:00.590827 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:55:00.590788    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22742942_2c3a_4d4b_b354_8cf1476abc4e.slice/crio-1fca116717a98bca86993b885f02065dcb0fdc7cc382fd56269854f20bf136af WatchSource:0}: Error finding container 1fca116717a98bca86993b885f02065dcb0fdc7cc382fd56269854f20bf136af: Status 404 returned error can't find the container with id 1fca116717a98bca86993b885f02065dcb0fdc7cc382fd56269854f20bf136af
May 20 13:55:00.604222 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:00.604192    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mtp2c"]
May 20 13:55:00.607058 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:55:00.607030    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a72c68_134e_4e0e_a8e7_e278d52ef7dc.slice/crio-00d19e21f4a162a7fd9f0259a4d81f6eadaa48d10cc4a8e5f28f711a60d9d92e WatchSource:0}: Error finding container 00d19e21f4a162a7fd9f0259a4d81f6eadaa48d10cc4a8e5f28f711a60d9d92e: Status 404 returned error can't find the container with id 00d19e21f4a162a7fd9f0259a4d81f6eadaa48d10cc4a8e5f28f711a60d9d92e
May 20 13:55:01.133882 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:01.133823    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtp2c" event={"ID":"17a72c68-134e-4e0e-a8e7-e278d52ef7dc","Type":"ContainerStarted","Data":"00d19e21f4a162a7fd9f0259a4d81f6eadaa48d10cc4a8e5f28f711a60d9d92e"}
May 20 13:55:01.135986 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:01.135956    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gngpj" event={"ID":"22742942-2c3a-4d4b-b354-8cf1476abc4e","Type":"ContainerStarted","Data":"1fca116717a98bca86993b885f02065dcb0fdc7cc382fd56269854f20bf136af"}
May 20 13:55:03.143761 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:03.143719    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtp2c" event={"ID":"17a72c68-134e-4e0e-a8e7-e278d52ef7dc","Type":"ContainerStarted","Data":"56c4600c5ac364edcc82e57415c58b9b908f9eca563082662f9f2404b9e6307e"}
May 20 13:55:03.144230 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:03.143770    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtp2c" event={"ID":"17a72c68-134e-4e0e-a8e7-e278d52ef7dc","Type":"ContainerStarted","Data":"7f80061c8295a52534d169a0d599445b5ddf4174965bc11456b42cc49db3f0ad"}
May 20 13:55:03.144230 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:03.143816    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mtp2c"
May 20 13:55:03.145129 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:03.145105    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gngpj" event={"ID":"22742942-2c3a-4d4b-b354-8cf1476abc4e","Type":"ContainerStarted","Data":"33bb67943f6d1574972f4a655e3f2db2ff33ba2bfe786cb0a81ee82d5ed88579"}
May 20 13:55:03.166162 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:03.166110    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-mtp2c" podStartSLOduration=251.302791093 podStartE2EDuration="4m13.166095607s" podCreationTimestamp="2026-05-20 13:50:50 +0000 UTC" firstStartedPulling="2026-05-20 13:55:00.608820194 +0000 UTC m=+285.231545240" lastFinishedPulling="2026-05-20 13:55:02.472124701 +0000 UTC m=+287.094849754" observedRunningTime="2026-05-20 13:55:03.16480935 +0000 UTC m=+287.787534420" watchObservedRunningTime="2026-05-20 13:55:03.166095607 +0000 UTC m=+287.788820675"
May 20 13:55:03.181115 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:03.181063    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gngpj" podStartSLOduration=251.297618628 podStartE2EDuration="4m13.181048028s" podCreationTimestamp="2026-05-20 13:50:50 +0000 UTC" firstStartedPulling="2026-05-20 13:55:00.592532334 +0000 UTC m=+285.215257380" lastFinishedPulling="2026-05-20 13:55:02.475961731 +0000 UTC m=+287.098686780" observedRunningTime="2026-05-20 13:55:03.180060761 +0000 UTC m=+287.802785823" watchObservedRunningTime="2026-05-20 13:55:03.181048028 +0000 UTC m=+287.803773096"
May 20 13:55:07.403793 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:07.403750    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:55:07.404224 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:07.403860    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:55:07.408361 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:07.408341    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:55:08.164246 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:08.164217    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:55:08.213589 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:08.213551    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585fdf4544-ltdpt"]
May 20 13:55:13.151345 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:13.151316    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mtp2c"
May 20 13:55:15.872594 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:15.868266    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 13:55:15.873396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:15.873371    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 13:55:15.882123 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:15.882099    2554 kubelet.go:1628] "Image garbage collection succeeded"
May 20 13:55:33.233881 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.233800    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-585fdf4544-ltdpt" podUID="8a678977-b24d-402a-8612-c2490f78dc6b" containerName="console" containerID="cri-o://4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd" gracePeriod=15
May 20 13:55:33.467136 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.467108    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-585fdf4544-ltdpt_8a678977-b24d-402a-8612-c2490f78dc6b/console/0.log"
May 20 13:55:33.467242 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.467169    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:55:33.552730 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.552630    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-oauth-config\") pod \"8a678977-b24d-402a-8612-c2490f78dc6b\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") "
May 20 13:55:33.552730 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.552686    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph7dx\" (UniqueName: \"kubernetes.io/projected/8a678977-b24d-402a-8612-c2490f78dc6b-kube-api-access-ph7dx\") pod \"8a678977-b24d-402a-8612-c2490f78dc6b\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") "
May 20 13:55:33.553019 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.552757    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-oauth-serving-cert\") pod \"8a678977-b24d-402a-8612-c2490f78dc6b\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") "
May 20 13:55:33.553019 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.552782    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-service-ca\") pod \"8a678977-b24d-402a-8612-c2490f78dc6b\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") "
May 20 13:55:33.553019 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.552808    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-console-config\") pod \"8a678977-b24d-402a-8612-c2490f78dc6b\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") "
May 20 13:55:33.553019 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.552882    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-trusted-ca-bundle\") pod \"8a678977-b24d-402a-8612-c2490f78dc6b\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") "
May 20 13:55:33.553019 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.552928    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-serving-cert\") pod \"8a678977-b24d-402a-8612-c2490f78dc6b\" (UID: \"8a678977-b24d-402a-8612-c2490f78dc6b\") "
May 20 13:55:33.553265 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.553213    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a678977-b24d-402a-8612-c2490f78dc6b" (UID: "8a678977-b24d-402a-8612-c2490f78dc6b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:55:33.553265 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.553222    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a678977-b24d-402a-8612-c2490f78dc6b" (UID: "8a678977-b24d-402a-8612-c2490f78dc6b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:55:33.553265 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.553243    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-console-config" (OuterVolumeSpecName: "console-config") pod "8a678977-b24d-402a-8612-c2490f78dc6b" (UID: "8a678977-b24d-402a-8612-c2490f78dc6b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:55:33.553417 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.553299    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a678977-b24d-402a-8612-c2490f78dc6b" (UID: "8a678977-b24d-402a-8612-c2490f78dc6b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:55:33.554946 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.554915    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a678977-b24d-402a-8612-c2490f78dc6b-kube-api-access-ph7dx" (OuterVolumeSpecName: "kube-api-access-ph7dx") pod "8a678977-b24d-402a-8612-c2490f78dc6b" (UID: "8a678977-b24d-402a-8612-c2490f78dc6b"). InnerVolumeSpecName "kube-api-access-ph7dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:55:33.555053 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.554924    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a678977-b24d-402a-8612-c2490f78dc6b" (UID: "8a678977-b24d-402a-8612-c2490f78dc6b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:55:33.555110 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.555082    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a678977-b24d-402a-8612-c2490f78dc6b" (UID: "8a678977-b24d-402a-8612-c2490f78dc6b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:55:33.654018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.653973    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-oauth-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:55:33.654018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.654013    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:55:33.654018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.654026    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-console-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:55:33.654260 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.654039    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a678977-b24d-402a-8612-c2490f78dc6b-trusted-ca-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:55:33.654260 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.654051    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:55:33.654260 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.654065    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a678977-b24d-402a-8612-c2490f78dc6b-console-oauth-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:55:33.654260 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:33.654077    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7dx\" (UniqueName: \"kubernetes.io/projected/8a678977-b24d-402a-8612-c2490f78dc6b-kube-api-access-ph7dx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:55:34.232500 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.232469    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-585fdf4544-ltdpt_8a678977-b24d-402a-8612-c2490f78dc6b/console/0.log"
May 20 13:55:34.232670 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.232516    2554 generic.go:358] "Generic (PLEG): container finished" podID="8a678977-b24d-402a-8612-c2490f78dc6b" containerID="4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd" exitCode=2
May 20 13:55:34.232670 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.232547    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fdf4544-ltdpt" event={"ID":"8a678977-b24d-402a-8612-c2490f78dc6b","Type":"ContainerDied","Data":"4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd"}
May 20 13:55:34.232670 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.232584    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585fdf4544-ltdpt"
May 20 13:55:34.232670 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.232593    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585fdf4544-ltdpt" event={"ID":"8a678977-b24d-402a-8612-c2490f78dc6b","Type":"ContainerDied","Data":"adf830439ce60f20f41046b4b73ce443b76ce85142cfa0ee0ca9e30481af437b"}
May 20 13:55:34.232670 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.232610    2554 scope.go:117] "RemoveContainer" containerID="4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd"
May 20 13:55:34.240624 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.240385    2554 scope.go:117] "RemoveContainer" containerID="4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd"
May 20 13:55:34.240877 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:55:34.240683    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd\": container with ID starting with 4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd not found: ID does not exist" containerID="4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd"
May 20 13:55:34.240877 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.240715    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd"} err="failed to get container status \"4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd\": rpc error: code = NotFound desc = could not find container \"4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd\": container with ID starting with 4f1f8ed4d3f21265ae446c22a6b6973c36552a93ad3f1e3f6684ddb455f782dd not found: ID does not exist"
May 20 13:55:34.250460 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.250435    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585fdf4544-ltdpt"]
May 20 13:55:34.255977 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:34.255956    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-585fdf4544-ltdpt"]
May 20 13:55:36.178995 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:55:36.178959    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a678977-b24d-402a-8612-c2490f78dc6b" path="/var/lib/kubelet/pods/8a678977-b24d-402a-8612-c2490f78dc6b/volumes"
May 20 13:56:08.128912 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.128811    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7457dcf47b-gxczx"]
May 20 13:56:08.129781 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.129752    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a678977-b24d-402a-8612-c2490f78dc6b" containerName="console"
May 20 13:56:08.129781 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.129777    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a678977-b24d-402a-8612-c2490f78dc6b" containerName="console"
May 20 13:56:08.129980 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.129909    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="8a678977-b24d-402a-8612-c2490f78dc6b" containerName="console"
May 20 13:56:08.132029 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.132001    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.144802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.144772    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7457dcf47b-gxczx"]
May 20 13:56:08.221274 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.221234    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjz6\" (UniqueName: \"kubernetes.io/projected/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-kube-api-access-tsjz6\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.221454 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.221289    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-oauth-config\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.221454 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.221308    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-oauth-serving-cert\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.221454 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.221386    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-service-ca\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.221454 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.221418    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-serving-cert\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.221608 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.221453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-trusted-ca-bundle\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.221608 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.221494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-config\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.322393 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.322346    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-serving-cert\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.322393 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.322398    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-trusted-ca-bundle\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.322654 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.322432    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-config\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.322654 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.322463    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjz6\" (UniqueName: \"kubernetes.io/projected/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-kube-api-access-tsjz6\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.322654 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.322508    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-oauth-config\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.322654 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.322533    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-oauth-serving-cert\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.322654 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.322586    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-service-ca\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.323360 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.323333    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-oauth-serving-cert\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.323457 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.323360    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-config\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.323529 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.323482    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-service-ca\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.323529 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.323496    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-trusted-ca-bundle\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.324856 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.324818    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-oauth-config\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.325025 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.325009    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-serving-cert\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.332143 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.332114    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjz6\" (UniqueName: \"kubernetes.io/projected/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-kube-api-access-tsjz6\") pod \"console-7457dcf47b-gxczx\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") " pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.441741 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.441697    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:08.568726 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.568700    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7457dcf47b-gxczx"]
May 20 13:56:08.571486 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:56:08.571457    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffe2955_cb91_4abc_9b1d_ad5b1c113980.slice/crio-ef4bff60cb7be75924edfe389238639645914f4cf60de872ed24bd2baf6dd2aa WatchSource:0}: Error finding container ef4bff60cb7be75924edfe389238639645914f4cf60de872ed24bd2baf6dd2aa: Status 404 returned error can't find the container with id ef4bff60cb7be75924edfe389238639645914f4cf60de872ed24bd2baf6dd2aa
May 20 13:56:08.573308 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:08.573291    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:56:09.337343 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:09.337310    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457dcf47b-gxczx" event={"ID":"1ffe2955-cb91-4abc-9b1d-ad5b1c113980","Type":"ContainerStarted","Data":"d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d"}
May 20 13:56:09.337691 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:09.337349    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457dcf47b-gxczx" event={"ID":"1ffe2955-cb91-4abc-9b1d-ad5b1c113980","Type":"ContainerStarted","Data":"ef4bff60cb7be75924edfe389238639645914f4cf60de872ed24bd2baf6dd2aa"}
May 20 13:56:09.362507 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:09.362444    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7457dcf47b-gxczx" podStartSLOduration=1.362423503 podStartE2EDuration="1.362423503s" podCreationTimestamp="2026-05-20 13:56:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:56:09.361202121 +0000 UTC m=+353.983927191" watchObservedRunningTime="2026-05-20 13:56:09.362423503 +0000 UTC m=+353.985148572"
May 20 13:56:18.441950 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:18.441912    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:18.441950 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:18.441953    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:18.446676 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:18.446652    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:19.375487 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:19.375457    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 13:56:19.424555 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:19.424523    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd74dd467-l9hsc"]
May 20 13:56:44.445001 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.444958    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7cd74dd467-l9hsc" podUID="eecdb085-eea1-4341-a932-ea1eab1921df" containerName="console" containerID="cri-o://b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616" gracePeriod=15
May 20 13:56:44.682419 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.682394    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cd74dd467-l9hsc_eecdb085-eea1-4341-a932-ea1eab1921df/console/0.log"
May 20 13:56:44.682549 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.682454    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:56:44.827754 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.827650    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb6n8\" (UniqueName: \"kubernetes.io/projected/eecdb085-eea1-4341-a932-ea1eab1921df-kube-api-access-xb6n8\") pod \"eecdb085-eea1-4341-a932-ea1eab1921df\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") "
May 20 13:56:44.827754 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.827724    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-console-config\") pod \"eecdb085-eea1-4341-a932-ea1eab1921df\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") "
May 20 13:56:44.827754 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.827760    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-serving-cert\") pod \"eecdb085-eea1-4341-a932-ea1eab1921df\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") "
May 20 13:56:44.828021 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.827885    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-service-ca\") pod \"eecdb085-eea1-4341-a932-ea1eab1921df\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") "
May 20 13:56:44.828021 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.827922    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-oauth-serving-cert\") pod \"eecdb085-eea1-4341-a932-ea1eab1921df\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") "
May 20 13:56:44.828021 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.827954    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-oauth-config\") pod \"eecdb085-eea1-4341-a932-ea1eab1921df\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") "
May 20 13:56:44.828021 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.827997    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-trusted-ca-bundle\") pod \"eecdb085-eea1-4341-a932-ea1eab1921df\" (UID: \"eecdb085-eea1-4341-a932-ea1eab1921df\") "
May 20 13:56:44.828526 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.828268    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-console-config" (OuterVolumeSpecName: "console-config") pod "eecdb085-eea1-4341-a932-ea1eab1921df" (UID: "eecdb085-eea1-4341-a932-ea1eab1921df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:56:44.828526 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.828301    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eecdb085-eea1-4341-a932-ea1eab1921df" (UID: "eecdb085-eea1-4341-a932-ea1eab1921df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:56:44.828526 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.828478    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-service-ca" (OuterVolumeSpecName: "service-ca") pod "eecdb085-eea1-4341-a932-ea1eab1921df" (UID: "eecdb085-eea1-4341-a932-ea1eab1921df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:56:44.828746 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.828529    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eecdb085-eea1-4341-a932-ea1eab1921df" (UID: "eecdb085-eea1-4341-a932-ea1eab1921df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:56:44.829949 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.829928    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eecdb085-eea1-4341-a932-ea1eab1921df" (UID: "eecdb085-eea1-4341-a932-ea1eab1921df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:56:44.830035 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.830006    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eecdb085-eea1-4341-a932-ea1eab1921df" (UID: "eecdb085-eea1-4341-a932-ea1eab1921df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:56:44.830035 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.830027    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecdb085-eea1-4341-a932-ea1eab1921df-kube-api-access-xb6n8" (OuterVolumeSpecName: "kube-api-access-xb6n8") pod "eecdb085-eea1-4341-a932-ea1eab1921df" (UID: "eecdb085-eea1-4341-a932-ea1eab1921df"). InnerVolumeSpecName "kube-api-access-xb6n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:56:44.933262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.928984    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb6n8\" (UniqueName: \"kubernetes.io/projected/eecdb085-eea1-4341-a932-ea1eab1921df-kube-api-access-xb6n8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:56:44.933262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.929055    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-console-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:56:44.933262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.929072    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:56:44.933262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.929087    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:56:44.933262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.929102    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-oauth-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:56:44.933262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.929120    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eecdb085-eea1-4341-a932-ea1eab1921df-console-oauth-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:56:44.933262 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:44.929134    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eecdb085-eea1-4341-a932-ea1eab1921df-trusted-ca-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:56:45.443190 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.443162    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cd74dd467-l9hsc_eecdb085-eea1-4341-a932-ea1eab1921df/console/0.log"
May 20 13:56:45.443363 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.443203    2554 generic.go:358] "Generic (PLEG): container finished" podID="eecdb085-eea1-4341-a932-ea1eab1921df" containerID="b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616" exitCode=2
May 20 13:56:45.443363 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.443236    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd74dd467-l9hsc" event={"ID":"eecdb085-eea1-4341-a932-ea1eab1921df","Type":"ContainerDied","Data":"b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616"}
May 20 13:56:45.443363 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.443293    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd74dd467-l9hsc" event={"ID":"eecdb085-eea1-4341-a932-ea1eab1921df","Type":"ContainerDied","Data":"990794310bb548cc4623f2aa45ec8c05235cbe025c6e7fa0468a599b544f6301"}
May 20 13:56:45.443363 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.443314    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd74dd467-l9hsc"
May 20 13:56:45.443531 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.443316    2554 scope.go:117] "RemoveContainer" containerID="b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616"
May 20 13:56:45.451706 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.451561    2554 scope.go:117] "RemoveContainer" containerID="b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616"
May 20 13:56:45.451953 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:56:45.451823    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616\": container with ID starting with b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616 not found: ID does not exist" containerID="b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616"
May 20 13:56:45.451953 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.451870    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616"} err="failed to get container status \"b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616\": rpc error: code = NotFound desc = could not find container \"b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616\": container with ID starting with b81dfa850158b8639a45cbca88721214b147acfdb5e2b89c731e336c41117616 not found: ID does not exist"
May 20 13:56:45.463188 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.463156    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd74dd467-l9hsc"]
May 20 13:56:45.466812 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:45.466789    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cd74dd467-l9hsc"]
May 20 13:56:46.175574 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:56:46.175530    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eecdb085-eea1-4341-a932-ea1eab1921df" path="/var/lib/kubelet/pods/eecdb085-eea1-4341-a932-ea1eab1921df/volumes"
May 20 13:57:25.633634 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.633593    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"]
May 20 13:57:25.634058 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.633911    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eecdb085-eea1-4341-a932-ea1eab1921df" containerName="console"
May 20 13:57:25.634058 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.633922    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eecdb085-eea1-4341-a932-ea1eab1921df" containerName="console"
May 20 13:57:25.634058 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.633969    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="eecdb085-eea1-4341-a932-ea1eab1921df" containerName="console"
May 20 13:57:25.636695 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.636673    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.639109 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.639082    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 13:57:25.639109 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.639101    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 13:57:25.639971 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.639952    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bm9kh\""
May 20 13:57:25.645817 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.645794    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"]
May 20 13:57:25.765514 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.765473    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvsj\" (UniqueName: \"kubernetes.io/projected/d94297ba-660b-40e5-ab9b-ac8384c07a0f-kube-api-access-7kvsj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.765699 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.765529    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.765699 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.765563    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.866960 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.866920    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvsj\" (UniqueName: \"kubernetes.io/projected/d94297ba-660b-40e5-ab9b-ac8384c07a0f-kube-api-access-7kvsj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.867132 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.866977    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.867132 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.867008    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.867451 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.867430    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.867489 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.867448    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.875714 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.875679    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvsj\" (UniqueName: \"kubernetes.io/projected/d94297ba-660b-40e5-ab9b-ac8384c07a0f-kube-api-access-7kvsj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:25.945808 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:25.945768    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:26.071256 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:26.071082    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"]
May 20 13:57:26.073910 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:57:26.073878    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94297ba_660b_40e5_ab9b_ac8384c07a0f.slice/crio-f2897c08b1c0b150a41e5eca34f0ca60d73975f30c78a4b3da3a485b9ad2b961 WatchSource:0}: Error finding container f2897c08b1c0b150a41e5eca34f0ca60d73975f30c78a4b3da3a485b9ad2b961: Status 404 returned error can't find the container with id f2897c08b1c0b150a41e5eca34f0ca60d73975f30c78a4b3da3a485b9ad2b961
May 20 13:57:26.561629 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:26.561584    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" event={"ID":"d94297ba-660b-40e5-ab9b-ac8384c07a0f","Type":"ContainerStarted","Data":"f2897c08b1c0b150a41e5eca34f0ca60d73975f30c78a4b3da3a485b9ad2b961"}
May 20 13:57:31.581524 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:31.581484    2554 generic.go:358] "Generic (PLEG): container finished" podID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerID="a7f3d501ddda6aab11a0e4b6b6e68f8cfa7cace31d60dc2bfb4e46b09ac159d6" exitCode=0
May 20 13:57:31.581941 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:31.581561    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" event={"ID":"d94297ba-660b-40e5-ab9b-ac8384c07a0f","Type":"ContainerDied","Data":"a7f3d501ddda6aab11a0e4b6b6e68f8cfa7cace31d60dc2bfb4e46b09ac159d6"}
May 20 13:57:33.589154 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:33.589117    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" event={"ID":"d94297ba-660b-40e5-ab9b-ac8384c07a0f","Type":"ContainerStarted","Data":"15cb113089d48ba2d2490d38280772222347a12191782aae249e043bc08107c9"}
May 20 13:57:34.594733 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:34.594696    2554 generic.go:358] "Generic (PLEG): container finished" podID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerID="15cb113089d48ba2d2490d38280772222347a12191782aae249e043bc08107c9" exitCode=0
May 20 13:57:34.595148 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:34.594756    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" event={"ID":"d94297ba-660b-40e5-ab9b-ac8384c07a0f","Type":"ContainerDied","Data":"15cb113089d48ba2d2490d38280772222347a12191782aae249e043bc08107c9"}
May 20 13:57:39.613370 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:39.613324    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" event={"ID":"d94297ba-660b-40e5-ab9b-ac8384c07a0f","Type":"ContainerStarted","Data":"6ca35758b8f4b2191b4f5ffdbc9de5729bbb4a678bb5bbc7c96d4d1c315c297e"}
May 20 13:57:39.631644 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:39.631590    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" podStartSLOduration=1.253410188 podStartE2EDuration="14.631574137s" podCreationTimestamp="2026-05-20 13:57:25 +0000 UTC" firstStartedPulling="2026-05-20 13:57:26.075632843 +0000 UTC m=+430.698357889" lastFinishedPulling="2026-05-20 13:57:39.453796789 +0000 UTC m=+444.076521838" observedRunningTime="2026-05-20 13:57:39.630035724 +0000 UTC m=+444.252760788" watchObservedRunningTime="2026-05-20 13:57:39.631574137 +0000 UTC m=+444.254299205"
May 20 13:57:40.617815 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:40.617777    2554 generic.go:358] "Generic (PLEG): container finished" podID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerID="6ca35758b8f4b2191b4f5ffdbc9de5729bbb4a678bb5bbc7c96d4d1c315c297e" exitCode=0
May 20 13:57:40.618214 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:40.617881    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" event={"ID":"d94297ba-660b-40e5-ab9b-ac8384c07a0f","Type":"ContainerDied","Data":"6ca35758b8f4b2191b4f5ffdbc9de5729bbb4a678bb5bbc7c96d4d1c315c297e"}
May 20 13:57:41.743062 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.743036    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:41.805883 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.805820    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-util\") pod \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") "
May 20 13:57:41.806051 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.805930    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kvsj\" (UniqueName: \"kubernetes.io/projected/d94297ba-660b-40e5-ab9b-ac8384c07a0f-kube-api-access-7kvsj\") pod \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") "
May 20 13:57:41.808150 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.808116    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94297ba-660b-40e5-ab9b-ac8384c07a0f-kube-api-access-7kvsj" (OuterVolumeSpecName: "kube-api-access-7kvsj") pod "d94297ba-660b-40e5-ab9b-ac8384c07a0f" (UID: "d94297ba-660b-40e5-ab9b-ac8384c07a0f"). InnerVolumeSpecName "kube-api-access-7kvsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:57:41.810908 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.810884    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-util" (OuterVolumeSpecName: "util") pod "d94297ba-660b-40e5-ab9b-ac8384c07a0f" (UID: "d94297ba-660b-40e5-ab9b-ac8384c07a0f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:57:41.906642 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.906610    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-bundle\") pod \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\" (UID: \"d94297ba-660b-40e5-ab9b-ac8384c07a0f\") "
May 20 13:57:41.906768 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.906753    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvsj\" (UniqueName: \"kubernetes.io/projected/d94297ba-660b-40e5-ab9b-ac8384c07a0f-kube-api-access-7kvsj\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:57:41.906768 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.906764    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:57:41.907197 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:41.907173    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-bundle" (OuterVolumeSpecName: "bundle") pod "d94297ba-660b-40e5-ab9b-ac8384c07a0f" (UID: "d94297ba-660b-40e5-ab9b-ac8384c07a0f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:57:42.007336 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:42.007296    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d94297ba-660b-40e5-ab9b-ac8384c07a0f-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:57:42.626068 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:42.625984    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk" event={"ID":"d94297ba-660b-40e5-ab9b-ac8384c07a0f","Type":"ContainerDied","Data":"f2897c08b1c0b150a41e5eca34f0ca60d73975f30c78a4b3da3a485b9ad2b961"}
May 20 13:57:42.626068 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:42.626020    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2897c08b1c0b150a41e5eca34f0ca60d73975f30c78a4b3da3a485b9ad2b961"
May 20 13:57:42.626068 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:42.626033    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chpxdk"
May 20 13:57:47.406298 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406264    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"]
May 20 13:57:47.406865 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406579    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerName="extract"
May 20 13:57:47.406865 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406593    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerName="extract"
May 20 13:57:47.406865 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406609    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerName="pull"
May 20 13:57:47.406865 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406614    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerName="pull"
May 20 13:57:47.406865 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406626    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerName="util"
May 20 13:57:47.406865 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406632    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerName="util"
May 20 13:57:47.406865 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.406680    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="d94297ba-660b-40e5-ab9b-ac8384c07a0f" containerName="extract"
May 20 13:57:47.447780 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.447735    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"]
May 20 13:57:47.447961 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.447801    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.450897 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.450867    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 20 13:57:47.450897 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.450883    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 20 13:57:47.451095 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.450882    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 20 13:57:47.451153 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.451137    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-9vp27\""
May 20 13:57:47.548860 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.548815    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3d7d0191-41ad-4675-bb19-a29f119793e7-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5\" (UID: \"3d7d0191-41ad-4675-bb19-a29f119793e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.549030 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.548994    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59c8\" (UniqueName: \"kubernetes.io/projected/3d7d0191-41ad-4675-bb19-a29f119793e7-kube-api-access-l59c8\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5\" (UID: \"3d7d0191-41ad-4675-bb19-a29f119793e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.649910 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.649876    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l59c8\" (UniqueName: \"kubernetes.io/projected/3d7d0191-41ad-4675-bb19-a29f119793e7-kube-api-access-l59c8\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5\" (UID: \"3d7d0191-41ad-4675-bb19-a29f119793e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.649910 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.649913    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3d7d0191-41ad-4675-bb19-a29f119793e7-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5\" (UID: \"3d7d0191-41ad-4675-bb19-a29f119793e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.652175 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.652151    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3d7d0191-41ad-4675-bb19-a29f119793e7-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5\" (UID: \"3d7d0191-41ad-4675-bb19-a29f119793e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.658303 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.658242    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59c8\" (UniqueName: \"kubernetes.io/projected/3d7d0191-41ad-4675-bb19-a29f119793e7-kube-api-access-l59c8\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5\" (UID: \"3d7d0191-41ad-4675-bb19-a29f119793e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.758338 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.758300    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:47.889662 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:47.889602    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"]
May 20 13:57:47.892829 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:57:47.892803    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7d0191_41ad_4675_bb19_a29f119793e7.slice/crio-1dd74e276ca0fd8fb845bf17c0a4d4ea504de27364641cf2f7478df7a0acd4ba WatchSource:0}: Error finding container 1dd74e276ca0fd8fb845bf17c0a4d4ea504de27364641cf2f7478df7a0acd4ba: Status 404 returned error can't find the container with id 1dd74e276ca0fd8fb845bf17c0a4d4ea504de27364641cf2f7478df7a0acd4ba
May 20 13:57:48.644479 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:48.644441    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5" event={"ID":"3d7d0191-41ad-4675-bb19-a29f119793e7","Type":"ContainerStarted","Data":"1dd74e276ca0fd8fb845bf17c0a4d4ea504de27364641cf2f7478df7a0acd4ba"}
May 20 13:57:54.213808 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.213765    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-j244n"]
May 20 13:57:54.216442 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.216417    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.219133 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.219108    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 20 13:57:54.219221 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.219160    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rxcft\""
May 20 13:57:54.219221 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.219116    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 20 13:57:54.225604 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.225574    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-j244n"]
May 20 13:57:54.311737 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.311686    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1b95c870-b754-45bb-8d24-e6d180a5a88c-cabundle0\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.311952 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.311925    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.312015 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.311974    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hf7g\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-kube-api-access-2hf7g\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.412656 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.412610    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.412830 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.412665    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2hf7g\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-kube-api-access-2hf7g\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.412830 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.412715    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1b95c870-b754-45bb-8d24-e6d180a5a88c-cabundle0\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.412830 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.412764    2554 secret.go:281] references non-existent secret key: ca.crt
May 20 13:57:54.412830 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.412786    2554 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 13:57:54.412830 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.412796    2554 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-j244n: references non-existent secret key: ca.crt
May 20 13:57:54.413047 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.412859    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates podName:1b95c870-b754-45bb-8d24-e6d180a5a88c nodeName:}" failed. No retries permitted until 2026-05-20 13:57:54.912830582 +0000 UTC m=+459.535555640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates") pod "keda-operator-ffbb595cb-j244n" (UID: "1b95c870-b754-45bb-8d24-e6d180a5a88c") : references non-existent secret key: ca.crt
May 20 13:57:54.413536 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.413515    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1b95c870-b754-45bb-8d24-e6d180a5a88c-cabundle0\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.421342 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.421314    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hf7g\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-kube-api-access-2hf7g\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.480677 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.480583    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"]
May 20 13:57:54.483189 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.483166    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.485736 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.485711    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 20 13:57:54.492857 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.492822    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"]
May 20 13:57:54.513961 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.513927    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.514135 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.514081    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bknn4\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-kube-api-access-bknn4\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.514190 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.514138    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f6662441-13d9-4ac8-9330-426e8c48b202-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.615211 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.615170    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f6662441-13d9-4ac8-9330-426e8c48b202-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.615412 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.615241    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.615412 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.615303    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bknn4\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-kube-api-access-bknn4\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.615537 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.615417    2554 secret.go:281] references non-existent secret key: tls.crt
May 20 13:57:54.615537 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.615444    2554 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 13:57:54.615537 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.615467    2554 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4: references non-existent secret key: tls.crt
May 20 13:57:54.615537 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.615533    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates podName:f6662441-13d9-4ac8-9330-426e8c48b202 nodeName:}" failed. No retries permitted until 2026-05-20 13:57:55.115512349 +0000 UTC m=+459.738237403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates") pod "keda-metrics-apiserver-7c9f485588-vzrg4" (UID: "f6662441-13d9-4ac8-9330-426e8c48b202") : references non-existent secret key: tls.crt
May 20 13:57:54.615723 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.615641    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f6662441-13d9-4ac8-9330-426e8c48b202-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.625599 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.625574    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bknn4\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-kube-api-access-bknn4\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:54.666569 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.666532    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5" event={"ID":"3d7d0191-41ad-4675-bb19-a29f119793e7","Type":"ContainerStarted","Data":"bf3dc21507e39c497a35e10b4bedd827a7d95d95faca07e7a43e1eb957f20d00"}
May 20 13:57:54.666748 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.666596    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:57:54.686931 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.686863    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5" podStartSLOduration=1.86534918 podStartE2EDuration="7.686830727s" podCreationTimestamp="2026-05-20 13:57:47 +0000 UTC" firstStartedPulling="2026-05-20 13:57:47.894481615 +0000 UTC m=+452.517206662" lastFinishedPulling="2026-05-20 13:57:53.715963157 +0000 UTC m=+458.338688209" observedRunningTime="2026-05-20 13:57:54.685236297 +0000 UTC m=+459.307961377" watchObservedRunningTime="2026-05-20 13:57:54.686830727 +0000 UTC m=+459.309555794"
May 20 13:57:54.919278 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:54.919229    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:54.919458 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.919379    2554 secret.go:281] references non-existent secret key: ca.crt
May 20 13:57:54.919458 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.919398    2554 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 13:57:54.919458 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.919407    2554 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-j244n: references non-existent secret key: ca.crt
May 20 13:57:54.919563 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:54.919474    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates podName:1b95c870-b754-45bb-8d24-e6d180a5a88c nodeName:}" failed. No retries permitted until 2026-05-20 13:57:55.91945574 +0000 UTC m=+460.542180797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates") pod "keda-operator-ffbb595cb-j244n" (UID: "1b95c870-b754-45bb-8d24-e6d180a5a88c") : references non-existent secret key: ca.crt
May 20 13:57:55.121349 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:55.121309    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:55.121527 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:55.121459    2554 secret.go:281] references non-existent secret key: tls.crt
May 20 13:57:55.121527 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:55.121477    2554 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 13:57:55.121527 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:55.121497    2554 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4: references non-existent secret key: tls.crt
May 20 13:57:55.121640 ip-10-0-141-251 kubenswrapper[2554]: E0520 13:57:55.121553    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates podName:f6662441-13d9-4ac8-9330-426e8c48b202 nodeName:}" failed. No retries permitted until 2026-05-20 13:57:56.121539771 +0000 UTC m=+460.744264817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates") pod "keda-metrics-apiserver-7c9f485588-vzrg4" (UID: "f6662441-13d9-4ac8-9330-426e8c48b202") : references non-existent secret key: tls.crt
May 20 13:57:55.928915 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:55.928877    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:55.931374 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:55.931350    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1b95c870-b754-45bb-8d24-e6d180a5a88c-certificates\") pod \"keda-operator-ffbb595cb-j244n\" (UID: \"1b95c870-b754-45bb-8d24-e6d180a5a88c\") " pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:56.027199 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.027154    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:57:56.131363 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.131326    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:56.134087 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.134054    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f6662441-13d9-4ac8-9330-426e8c48b202-certificates\") pod \"keda-metrics-apiserver-7c9f485588-vzrg4\" (UID: \"f6662441-13d9-4ac8-9330-426e8c48b202\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:56.155945 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.155916    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-j244n"]
May 20 13:57:56.158307 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:57:56.158277    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b95c870_b754_45bb_8d24_e6d180a5a88c.slice/crio-2383b28ed97f75b07f0651b7dddaf3911b26efa857cefc2cf401e7ea13a7052c WatchSource:0}: Error finding container 2383b28ed97f75b07f0651b7dddaf3911b26efa857cefc2cf401e7ea13a7052c: Status 404 returned error can't find the container with id 2383b28ed97f75b07f0651b7dddaf3911b26efa857cefc2cf401e7ea13a7052c
May 20 13:57:56.294420 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.294332    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:56.425533 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.425504    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"]
May 20 13:57:56.427457 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:57:56.427414    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6662441_13d9_4ac8_9330_426e8c48b202.slice/crio-3f341a888a53a4bd4c11f906ab3a24a61ec786d85dd1f68dccfb5167b6db4aa1 WatchSource:0}: Error finding container 3f341a888a53a4bd4c11f906ab3a24a61ec786d85dd1f68dccfb5167b6db4aa1: Status 404 returned error can't find the container with id 3f341a888a53a4bd4c11f906ab3a24a61ec786d85dd1f68dccfb5167b6db4aa1
May 20 13:57:56.674000 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.673966    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-j244n" event={"ID":"1b95c870-b754-45bb-8d24-e6d180a5a88c","Type":"ContainerStarted","Data":"2383b28ed97f75b07f0651b7dddaf3911b26efa857cefc2cf401e7ea13a7052c"}
May 20 13:57:56.674995 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:56.674970    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4" event={"ID":"f6662441-13d9-4ac8-9330-426e8c48b202","Type":"ContainerStarted","Data":"3f341a888a53a4bd4c11f906ab3a24a61ec786d85dd1f68dccfb5167b6db4aa1"}
May 20 13:57:59.687207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:59.687172    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4" event={"ID":"f6662441-13d9-4ac8-9330-426e8c48b202","Type":"ContainerStarted","Data":"cc15ad9add7f8dbe4d740d07eaf12ab219a3e3eb8ef953df296014ec05b48565"}
May 20 13:57:59.687634 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:59.687286    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:57:59.703563 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:57:59.703492    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4" podStartSLOduration=2.645086907 podStartE2EDuration="5.70347573s" podCreationTimestamp="2026-05-20 13:57:54 +0000 UTC" firstStartedPulling="2026-05-20 13:57:56.428875165 +0000 UTC m=+461.051600212" lastFinishedPulling="2026-05-20 13:57:59.487263979 +0000 UTC m=+464.109989035" observedRunningTime="2026-05-20 13:57:59.703296813 +0000 UTC m=+464.326021883" watchObservedRunningTime="2026-05-20 13:57:59.70347573 +0000 UTC m=+464.326200797"
May 20 13:58:01.694975 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:01.694938    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-j244n" event={"ID":"1b95c870-b754-45bb-8d24-e6d180a5a88c","Type":"ContainerStarted","Data":"2fa2b1d6ab276671477a3a99fc79384f3e7518f1a3e5abfc7be2495244395d52"}
May 20 13:58:01.695371 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:01.695001    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:58:01.713779 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:01.713731    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-j244n" podStartSLOduration=2.520267815 podStartE2EDuration="7.713716895s" podCreationTimestamp="2026-05-20 13:57:54 +0000 UTC" firstStartedPulling="2026-05-20 13:57:56.159581714 +0000 UTC m=+460.782306759" lastFinishedPulling="2026-05-20 13:58:01.353030791 +0000 UTC m=+465.975755839" observedRunningTime="2026-05-20 13:58:01.713028328 +0000 UTC m=+466.335753399" watchObservedRunningTime="2026-05-20 13:58:01.713716895 +0000 UTC m=+466.336441962"
May 20 13:58:10.695161 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:10.695122    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-vzrg4"
May 20 13:58:15.672122 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:15.672090    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-q6tr5"
May 20 13:58:22.700309 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:22.700279    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-j244n"
May 20 13:58:45.642793 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.642748    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"]
May 20 13:58:45.645297 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.645278    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.648817 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.648790    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 13:58:45.649748 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.649724    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 13:58:45.649872 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.649724    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bm9kh\""
May 20 13:58:45.664151 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.664117    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"]
May 20 13:58:45.740266 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.740229    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws759\" (UniqueName: \"kubernetes.io/projected/49621991-884c-47b6-8e62-197a760378c2-kube-api-access-ws759\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.740266 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.740278    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.740499 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.740343    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.841028 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.840994    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ws759\" (UniqueName: \"kubernetes.io/projected/49621991-884c-47b6-8e62-197a760378c2-kube-api-access-ws759\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.841218 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.841038    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.841218 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.841156    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.841378 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.841361    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.841443 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.841426    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.848983 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.848958    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws759\" (UniqueName: \"kubernetes.io/projected/49621991-884c-47b6-8e62-197a760378c2-kube-api-access-ws759\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:45.954859 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:45.954800    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:46.088740 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:46.088711    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"]
May 20 13:58:46.091160 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:58:46.091122    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49621991_884c_47b6_8e62_197a760378c2.slice/crio-e77c241052f14e3f2d8a0f1844ecf6745e6bc7a6dfd270c7b11c3d613b9e3b5e WatchSource:0}: Error finding container e77c241052f14e3f2d8a0f1844ecf6745e6bc7a6dfd270c7b11c3d613b9e3b5e: Status 404 returned error can't find the container with id e77c241052f14e3f2d8a0f1844ecf6745e6bc7a6dfd270c7b11c3d613b9e3b5e
May 20 13:58:46.842447 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:46.842412    2554 generic.go:358] "Generic (PLEG): container finished" podID="49621991-884c-47b6-8e62-197a760378c2" containerID="05ce647d4e3b721981798e99a0b221dc8b9e24e22f6aff2b37ddf41f4dfaf266" exitCode=0
May 20 13:58:46.842825 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:46.842461    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t" event={"ID":"49621991-884c-47b6-8e62-197a760378c2","Type":"ContainerDied","Data":"05ce647d4e3b721981798e99a0b221dc8b9e24e22f6aff2b37ddf41f4dfaf266"}
May 20 13:58:46.842825 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:46.842483    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t" event={"ID":"49621991-884c-47b6-8e62-197a760378c2","Type":"ContainerStarted","Data":"e77c241052f14e3f2d8a0f1844ecf6745e6bc7a6dfd270c7b11c3d613b9e3b5e"}
May 20 13:58:49.854504 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:49.854462    2554 generic.go:358] "Generic (PLEG): container finished" podID="49621991-884c-47b6-8e62-197a760378c2" containerID="a38c9c4adc01b1c9e4d6ad06b6f5e90052f40647e801e3c27a16de73ab5a26c6" exitCode=0
May 20 13:58:49.854935 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:49.854536    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t" event={"ID":"49621991-884c-47b6-8e62-197a760378c2","Type":"ContainerDied","Data":"a38c9c4adc01b1c9e4d6ad06b6f5e90052f40647e801e3c27a16de73ab5a26c6"}
May 20 13:58:50.859585 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:50.859545    2554 generic.go:358] "Generic (PLEG): container finished" podID="49621991-884c-47b6-8e62-197a760378c2" containerID="7f5d43d7292027357bd12b5aea4d37595de0dc1834011241e1c84534482b2a55" exitCode=0
May 20 13:58:50.860052 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:50.859625    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t" event={"ID":"49621991-884c-47b6-8e62-197a760378c2","Type":"ContainerDied","Data":"7f5d43d7292027357bd12b5aea4d37595de0dc1834011241e1c84534482b2a55"}
May 20 13:58:51.985896 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:51.985871    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:52.096066 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.096022    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-bundle\") pod \"49621991-884c-47b6-8e62-197a760378c2\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") "
May 20 13:58:52.096245 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.096114    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws759\" (UniqueName: \"kubernetes.io/projected/49621991-884c-47b6-8e62-197a760378c2-kube-api-access-ws759\") pod \"49621991-884c-47b6-8e62-197a760378c2\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") "
May 20 13:58:52.096245 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.096134    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-util\") pod \"49621991-884c-47b6-8e62-197a760378c2\" (UID: \"49621991-884c-47b6-8e62-197a760378c2\") "
May 20 13:58:52.096729 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.096697    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-bundle" (OuterVolumeSpecName: "bundle") pod "49621991-884c-47b6-8e62-197a760378c2" (UID: "49621991-884c-47b6-8e62-197a760378c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:58:52.098238 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.098211    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49621991-884c-47b6-8e62-197a760378c2-kube-api-access-ws759" (OuterVolumeSpecName: "kube-api-access-ws759") pod "49621991-884c-47b6-8e62-197a760378c2" (UID: "49621991-884c-47b6-8e62-197a760378c2"). InnerVolumeSpecName "kube-api-access-ws759". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:58:52.100605 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.100584    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-util" (OuterVolumeSpecName: "util") pod "49621991-884c-47b6-8e62-197a760378c2" (UID: "49621991-884c-47b6-8e62-197a760378c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:58:52.197546 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.197508    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:58:52.197546 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.197541    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws759\" (UniqueName: \"kubernetes.io/projected/49621991-884c-47b6-8e62-197a760378c2-kube-api-access-ws759\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:58:52.197546 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.197552    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49621991-884c-47b6-8e62-197a760378c2-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:58:52.873510 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.873473    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t" event={"ID":"49621991-884c-47b6-8e62-197a760378c2","Type":"ContainerDied","Data":"e77c241052f14e3f2d8a0f1844ecf6745e6bc7a6dfd270c7b11c3d613b9e3b5e"}
May 20 13:58:52.873510 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.873509    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77c241052f14e3f2d8a0f1844ecf6745e6bc7a6dfd270c7b11c3d613b9e3b5e"
May 20 13:58:52.873737 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:52.873515    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dxkq8t"
May 20 13:58:58.499626 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499548    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"]
May 20 13:58:58.500055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499900    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49621991-884c-47b6-8e62-197a760378c2" containerName="util"
May 20 13:58:58.500055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499914    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49621991-884c-47b6-8e62-197a760378c2" containerName="util"
May 20 13:58:58.500055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499927    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49621991-884c-47b6-8e62-197a760378c2" containerName="extract"
May 20 13:58:58.500055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499934    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49621991-884c-47b6-8e62-197a760378c2" containerName="extract"
May 20 13:58:58.500055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499944    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49621991-884c-47b6-8e62-197a760378c2" containerName="pull"
May 20 13:58:58.500055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499949    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49621991-884c-47b6-8e62-197a760378c2" containerName="pull"
May 20 13:58:58.500055 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.499992    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="49621991-884c-47b6-8e62-197a760378c2" containerName="extract"
May 20 13:58:58.502416 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.502395    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.505018 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.504992    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
May 20 13:58:58.505117 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.505084    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wjw7r\""
May 20 13:58:58.505117 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.505091    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
May 20 13:58:58.514492 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.514465    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"]
May 20 13:58:58.653793 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.653755    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p564\" (UniqueName: \"kubernetes.io/projected/d55b9782-8450-4eb4-8a07-37d691bddb9e-kube-api-access-5p564\") pod \"cert-manager-operator-controller-manager-54b9655956-wrsxp\" (UID: \"d55b9782-8450-4eb4-8a07-37d691bddb9e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.653793 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.653799    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d55b9782-8450-4eb4-8a07-37d691bddb9e-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-wrsxp\" (UID: \"d55b9782-8450-4eb4-8a07-37d691bddb9e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.754453 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.754348    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d55b9782-8450-4eb4-8a07-37d691bddb9e-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-wrsxp\" (UID: \"d55b9782-8450-4eb4-8a07-37d691bddb9e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.754632 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.754477    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5p564\" (UniqueName: \"kubernetes.io/projected/d55b9782-8450-4eb4-8a07-37d691bddb9e-kube-api-access-5p564\") pod \"cert-manager-operator-controller-manager-54b9655956-wrsxp\" (UID: \"d55b9782-8450-4eb4-8a07-37d691bddb9e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.754735 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.754715    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d55b9782-8450-4eb4-8a07-37d691bddb9e-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-wrsxp\" (UID: \"d55b9782-8450-4eb4-8a07-37d691bddb9e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.763355 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.763327    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p564\" (UniqueName: \"kubernetes.io/projected/d55b9782-8450-4eb4-8a07-37d691bddb9e-kube-api-access-5p564\") pod \"cert-manager-operator-controller-manager-54b9655956-wrsxp\" (UID: \"d55b9782-8450-4eb4-8a07-37d691bddb9e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.843753 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.843714    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"
May 20 13:58:58.981697 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:58.981661    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp"]
May 20 13:58:58.984286 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:58:58.984252    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55b9782_8450_4eb4_8a07_37d691bddb9e.slice/crio-076494766a64877febffaaec5c1953f67f44c67ebc56a5a272abb3ca136bf366 WatchSource:0}: Error finding container 076494766a64877febffaaec5c1953f67f44c67ebc56a5a272abb3ca136bf366: Status 404 returned error can't find the container with id 076494766a64877febffaaec5c1953f67f44c67ebc56a5a272abb3ca136bf366
May 20 13:58:59.899090 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:58:59.899045    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp" event={"ID":"d55b9782-8450-4eb4-8a07-37d691bddb9e","Type":"ContainerStarted","Data":"076494766a64877febffaaec5c1953f67f44c67ebc56a5a272abb3ca136bf366"}
May 20 13:59:01.913231 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:01.913196    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp" event={"ID":"d55b9782-8450-4eb4-8a07-37d691bddb9e","Type":"ContainerStarted","Data":"73650d2fe48e4dd42fa285b8cd444cf8cafabbadab94d1e070e7492b0bdb0348"}
May 20 13:59:01.937458 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:01.937397    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-wrsxp" podStartSLOduration=1.581727533 podStartE2EDuration="3.9373803s" podCreationTimestamp="2026-05-20 13:58:58 +0000 UTC" firstStartedPulling="2026-05-20 13:58:58.987791519 +0000 UTC m=+523.610516569" lastFinishedPulling="2026-05-20 13:59:01.343444281 +0000 UTC m=+525.966169336" observedRunningTime="2026-05-20 13:59:01.934725211 +0000 UTC m=+526.557450293" watchObservedRunningTime="2026-05-20 13:59:01.9373803 +0000 UTC m=+526.560105367"
May 20 13:59:07.009738 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.009654    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-d96pt"]
May 20 13:59:07.012169 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.012148    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.014547 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.014525    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 20 13:59:07.015241 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.015226    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-d2vgb\""
May 20 13:59:07.015304 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.015261    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 20 13:59:07.023802 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.023775    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-d96pt"]
May 20 13:59:07.128577 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.128536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77b0d36e-e787-4712-9141-e9914913b36a-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-d96pt\" (UID: \"77b0d36e-e787-4712-9141-e9914913b36a\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.128577 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.128581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdx9\" (UniqueName: \"kubernetes.io/projected/77b0d36e-e787-4712-9141-e9914913b36a-kube-api-access-tpdx9\") pod \"cert-manager-webhook-587ccfb98-d96pt\" (UID: \"77b0d36e-e787-4712-9141-e9914913b36a\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.229890 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.229823    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77b0d36e-e787-4712-9141-e9914913b36a-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-d96pt\" (UID: \"77b0d36e-e787-4712-9141-e9914913b36a\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.230036 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.229906    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdx9\" (UniqueName: \"kubernetes.io/projected/77b0d36e-e787-4712-9141-e9914913b36a-kube-api-access-tpdx9\") pod \"cert-manager-webhook-587ccfb98-d96pt\" (UID: \"77b0d36e-e787-4712-9141-e9914913b36a\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.238545 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.238515    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77b0d36e-e787-4712-9141-e9914913b36a-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-d96pt\" (UID: \"77b0d36e-e787-4712-9141-e9914913b36a\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.238663 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.238617    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdx9\" (UniqueName: \"kubernetes.io/projected/77b0d36e-e787-4712-9141-e9914913b36a-kube-api-access-tpdx9\") pod \"cert-manager-webhook-587ccfb98-d96pt\" (UID: \"77b0d36e-e787-4712-9141-e9914913b36a\") " pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.329518 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.329427    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:07.460384 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.460196    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-d96pt"]
May 20 13:59:07.462637 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:59:07.462607    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77b0d36e_e787_4712_9141_e9914913b36a.slice/crio-dc991f47862c9c81cd04b3567eee62ed32a1e26335d8f586bb2ba2dcbd3ca4c4 WatchSource:0}: Error finding container dc991f47862c9c81cd04b3567eee62ed32a1e26335d8f586bb2ba2dcbd3ca4c4: Status 404 returned error can't find the container with id dc991f47862c9c81cd04b3567eee62ed32a1e26335d8f586bb2ba2dcbd3ca4c4
May 20 13:59:07.751126 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.751088    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"]
May 20 13:59:07.754303 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.754281    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.756827 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.756798    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 13:59:07.756951 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.756806    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 13:59:07.757522 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.757503    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bm9kh\""
May 20 13:59:07.761636 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.761614    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"]
May 20 13:59:07.835639 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.835602    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.835790 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.835657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.835790 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.835694    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46p9z\" (UniqueName: \"kubernetes.io/projected/1b55f022-7327-4f39-a413-8726a1e05f25-kube-api-access-46p9z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.934727 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.934691    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt" event={"ID":"77b0d36e-e787-4712-9141-e9914913b36a","Type":"ContainerStarted","Data":"dc991f47862c9c81cd04b3567eee62ed32a1e26335d8f586bb2ba2dcbd3ca4c4"}
May 20 13:59:07.937152 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.937131    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.937231 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.937170    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-46p9z\" (UniqueName: \"kubernetes.io/projected/1b55f022-7327-4f39-a413-8726a1e05f25-kube-api-access-46p9z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.937266 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.937248    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.937630 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.937612    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.937669 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.937612    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:07.945795 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:07.945762    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p9z\" (UniqueName: \"kubernetes.io/projected/1b55f022-7327-4f39-a413-8726a1e05f25-kube-api-access-46p9z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:08.064972 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:08.064884    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:08.201968 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:08.201869    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"]
May 20 13:59:08.204585 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:59:08.204556    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b55f022_7327_4f39_a413_8726a1e05f25.slice/crio-71a6754e08bf6950e34beb6dd5a90786d8c6b163f7e243d02b24cb383194c3ac WatchSource:0}: Error finding container 71a6754e08bf6950e34beb6dd5a90786d8c6b163f7e243d02b24cb383194c3ac: Status 404 returned error can't find the container with id 71a6754e08bf6950e34beb6dd5a90786d8c6b163f7e243d02b24cb383194c3ac
May 20 13:59:08.939657 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:08.939619    2554 generic.go:358] "Generic (PLEG): container finished" podID="1b55f022-7327-4f39-a413-8726a1e05f25" containerID="07a12e54a350521ce798600dbea70329707464c75791cc0e33b0b52796e618d0" exitCode=0
May 20 13:59:08.939860 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:08.939690    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml" event={"ID":"1b55f022-7327-4f39-a413-8726a1e05f25","Type":"ContainerDied","Data":"07a12e54a350521ce798600dbea70329707464c75791cc0e33b0b52796e618d0"}
May 20 13:59:08.939860 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:08.939736    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml" event={"ID":"1b55f022-7327-4f39-a413-8726a1e05f25","Type":"ContainerStarted","Data":"71a6754e08bf6950e34beb6dd5a90786d8c6b163f7e243d02b24cb383194c3ac"}
May 20 13:59:12.956734 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:12.956699    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt" event={"ID":"77b0d36e-e787-4712-9141-e9914913b36a","Type":"ContainerStarted","Data":"2ab73852908969ec1c84b601d0ad616ad0c179c4c0449b88d1f9e91586d1a4fa"}
May 20 13:59:12.957203 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:12.956769    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:12.958327 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:12.958299    2554 generic.go:358] "Generic (PLEG): container finished" podID="1b55f022-7327-4f39-a413-8726a1e05f25" containerID="31a0a8374cf7739ae1943a969a16bd41fc40ae7ce929af43528c8008b4fd08f6" exitCode=0
May 20 13:59:12.958439 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:12.958333    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml" event={"ID":"1b55f022-7327-4f39-a413-8726a1e05f25","Type":"ContainerDied","Data":"31a0a8374cf7739ae1943a969a16bd41fc40ae7ce929af43528c8008b4fd08f6"}
May 20 13:59:12.974634 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:12.974578    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt" podStartSLOduration=2.343159418 podStartE2EDuration="6.974558546s" podCreationTimestamp="2026-05-20 13:59:06 +0000 UTC" firstStartedPulling="2026-05-20 13:59:07.46440996 +0000 UTC m=+532.087135006" lastFinishedPulling="2026-05-20 13:59:12.095809086 +0000 UTC m=+536.718534134" observedRunningTime="2026-05-20 13:59:12.972374096 +0000 UTC m=+537.595099238" watchObservedRunningTime="2026-05-20 13:59:12.974558546 +0000 UTC m=+537.597283615"
May 20 13:59:13.963442 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:13.963406    2554 generic.go:358] "Generic (PLEG): container finished" podID="1b55f022-7327-4f39-a413-8726a1e05f25" containerID="135cca205dd46dc03fa4710bb7d122e4b2709a77c6f1ceb69a4ac3e73a47f809" exitCode=0
May 20 13:59:13.963828 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:13.963500    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml" event={"ID":"1b55f022-7327-4f39-a413-8726a1e05f25","Type":"ContainerDied","Data":"135cca205dd46dc03fa4710bb7d122e4b2709a77c6f1ceb69a4ac3e73a47f809"}
May 20 13:59:15.093514 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.093485    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:15.208831 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.208795    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46p9z\" (UniqueName: \"kubernetes.io/projected/1b55f022-7327-4f39-a413-8726a1e05f25-kube-api-access-46p9z\") pod \"1b55f022-7327-4f39-a413-8726a1e05f25\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") "
May 20 13:59:15.209012 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.208891    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-util\") pod \"1b55f022-7327-4f39-a413-8726a1e05f25\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") "
May 20 13:59:15.209012 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.208915    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-bundle\") pod \"1b55f022-7327-4f39-a413-8726a1e05f25\" (UID: \"1b55f022-7327-4f39-a413-8726a1e05f25\") "
May 20 13:59:15.209325 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.209293    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-bundle" (OuterVolumeSpecName: "bundle") pod "1b55f022-7327-4f39-a413-8726a1e05f25" (UID: "1b55f022-7327-4f39-a413-8726a1e05f25"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:59:15.211110 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.211082    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b55f022-7327-4f39-a413-8726a1e05f25-kube-api-access-46p9z" (OuterVolumeSpecName: "kube-api-access-46p9z") pod "1b55f022-7327-4f39-a413-8726a1e05f25" (UID: "1b55f022-7327-4f39-a413-8726a1e05f25"). InnerVolumeSpecName "kube-api-access-46p9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:59:15.213333 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.213314    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-util" (OuterVolumeSpecName: "util") pod "1b55f022-7327-4f39-a413-8726a1e05f25" (UID: "1b55f022-7327-4f39-a413-8726a1e05f25"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:59:15.310072 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.309984    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46p9z\" (UniqueName: \"kubernetes.io/projected/1b55f022-7327-4f39-a413-8726a1e05f25-kube-api-access-46p9z\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:59:15.310072 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.310017    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:59:15.310072 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.310028    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b55f022-7327-4f39-a413-8726a1e05f25-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:59:15.971832 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.971789    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml" event={"ID":"1b55f022-7327-4f39-a413-8726a1e05f25","Type":"ContainerDied","Data":"71a6754e08bf6950e34beb6dd5a90786d8c6b163f7e243d02b24cb383194c3ac"}
May 20 13:59:15.971832 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.971824    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a6754e08bf6950e34beb6dd5a90786d8c6b163f7e243d02b24cb383194c3ac"
May 20 13:59:15.972052 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:15.971855    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f56wml"
May 20 13:59:18.966157 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:18.966125    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-d96pt"
May 20 13:59:22.768346 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768312    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-789hc"]
May 20 13:59:22.768707 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768632    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55f022-7327-4f39-a413-8726a1e05f25" containerName="util"
May 20 13:59:22.768707 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768643    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55f022-7327-4f39-a413-8726a1e05f25" containerName="util"
May 20 13:59:22.768707 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768650    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55f022-7327-4f39-a413-8726a1e05f25" containerName="pull"
May 20 13:59:22.768707 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768658    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55f022-7327-4f39-a413-8726a1e05f25" containerName="pull"
May 20 13:59:22.768707 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768679    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55f022-7327-4f39-a413-8726a1e05f25" containerName="extract"
May 20 13:59:22.768707 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768685    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55f022-7327-4f39-a413-8726a1e05f25" containerName="extract"
May 20 13:59:22.768930 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.768736    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="1b55f022-7327-4f39-a413-8726a1e05f25" containerName="extract"
May 20 13:59:22.804161 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.804115    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-789hc"]
May 20 13:59:22.804322 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.804243    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:22.806729 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.806703    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9krx5\""
May 20 13:59:22.870190 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.870149    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12117670-0563-4003-bb03-f8497dafd2bf-bound-sa-token\") pod \"cert-manager-79c8d999ff-789hc\" (UID: \"12117670-0563-4003-bb03-f8497dafd2bf\") " pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:22.870376 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.870222    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qt2t\" (UniqueName: \"kubernetes.io/projected/12117670-0563-4003-bb03-f8497dafd2bf-kube-api-access-9qt2t\") pod \"cert-manager-79c8d999ff-789hc\" (UID: \"12117670-0563-4003-bb03-f8497dafd2bf\") " pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:22.971471 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.971432    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12117670-0563-4003-bb03-f8497dafd2bf-bound-sa-token\") pod \"cert-manager-79c8d999ff-789hc\" (UID: \"12117670-0563-4003-bb03-f8497dafd2bf\") " pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:22.971707 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.971567    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9qt2t\" (UniqueName: \"kubernetes.io/projected/12117670-0563-4003-bb03-f8497dafd2bf-kube-api-access-9qt2t\") pod \"cert-manager-79c8d999ff-789hc\" (UID: \"12117670-0563-4003-bb03-f8497dafd2bf\") " pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:22.980461 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.980425    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12117670-0563-4003-bb03-f8497dafd2bf-bound-sa-token\") pod \"cert-manager-79c8d999ff-789hc\" (UID: \"12117670-0563-4003-bb03-f8497dafd2bf\") " pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:22.980589 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:22.980559    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qt2t\" (UniqueName: \"kubernetes.io/projected/12117670-0563-4003-bb03-f8497dafd2bf-kube-api-access-9qt2t\") pod \"cert-manager-79c8d999ff-789hc\" (UID: \"12117670-0563-4003-bb03-f8497dafd2bf\") " pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:23.113888 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:23.113756    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-789hc"
May 20 13:59:23.249587 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:23.249547    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-789hc"]
May 20 13:59:23.253240 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:59:23.253202    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12117670_0563_4003_bb03_f8497dafd2bf.slice/crio-1c402ccaed237773d30a1bf6b77739978a016c29c4f51c197dfb8662c1b045e6 WatchSource:0}: Error finding container 1c402ccaed237773d30a1bf6b77739978a016c29c4f51c197dfb8662c1b045e6: Status 404 returned error can't find the container with id 1c402ccaed237773d30a1bf6b77739978a016c29c4f51c197dfb8662c1b045e6
May 20 13:59:24.002062 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:24.002022    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-789hc" event={"ID":"12117670-0563-4003-bb03-f8497dafd2bf","Type":"ContainerStarted","Data":"f33ed345552763e82ca3e9bf2b1dee7762a9d36fcecaf0a8fb6a646d6100de11"}
May 20 13:59:24.002062 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:24.002064    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-789hc" event={"ID":"12117670-0563-4003-bb03-f8497dafd2bf","Type":"ContainerStarted","Data":"1c402ccaed237773d30a1bf6b77739978a016c29c4f51c197dfb8662c1b045e6"}
May 20 13:59:24.018308 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:24.018254    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-789hc" podStartSLOduration=2.018237893 podStartE2EDuration="2.018237893s" podCreationTimestamp="2026-05-20 13:59:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:59:24.016779478 +0000 UTC m=+548.639504547" watchObservedRunningTime="2026-05-20 13:59:24.018237893 +0000 UTC m=+548.640962961"
May 20 13:59:41.747231 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.747195    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"]
May 20 13:59:41.749870 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.749828    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.752242 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.752218    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 13:59:41.752998 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.752975    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 13:59:41.753105 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.752995    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bm9kh\""
May 20 13:59:41.758580 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.758546    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"]
May 20 13:59:41.818409 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.818357    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.818582 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.818425    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.818582 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.818464    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdn6d\" (UniqueName: \"kubernetes.io/projected/72404614-34dd-495d-9cbb-f0d0224422ac-kube-api-access-rdn6d\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.919858 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.919815    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.920049 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.919887    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.920049 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.919912    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn6d\" (UniqueName: \"kubernetes.io/projected/72404614-34dd-495d-9cbb-f0d0224422ac-kube-api-access-rdn6d\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.920228 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.920205    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.920291 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.920270    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:41.928799 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:41.928774    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn6d\" (UniqueName: \"kubernetes.io/projected/72404614-34dd-495d-9cbb-f0d0224422ac-kube-api-access-rdn6d\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:42.060314 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:42.060210    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:42.196518 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:42.196490    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"]
May 20 13:59:42.198581 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:59:42.198550    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72404614_34dd_495d_9cbb_f0d0224422ac.slice/crio-73c059ce28765e409b47cc60a636ea556f89455c67e448a54357a4d8ef594a83 WatchSource:0}: Error finding container 73c059ce28765e409b47cc60a636ea556f89455c67e448a54357a4d8ef594a83: Status 404 returned error can't find the container with id 73c059ce28765e409b47cc60a636ea556f89455c67e448a54357a4d8ef594a83
May 20 13:59:43.069207 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:43.069164    2554 generic.go:358] "Generic (PLEG): container finished" podID="72404614-34dd-495d-9cbb-f0d0224422ac" containerID="82d29215131e7bd24cd717ea9ea565f667c61c9ec36a920c0a15acb205b0dfaa" exitCode=0
May 20 13:59:43.069639 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:43.069249    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x" event={"ID":"72404614-34dd-495d-9cbb-f0d0224422ac","Type":"ContainerDied","Data":"82d29215131e7bd24cd717ea9ea565f667c61c9ec36a920c0a15acb205b0dfaa"}
May 20 13:59:43.069639 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:43.069291    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x" event={"ID":"72404614-34dd-495d-9cbb-f0d0224422ac","Type":"ContainerStarted","Data":"73c059ce28765e409b47cc60a636ea556f89455c67e448a54357a4d8ef594a83"}
May 20 13:59:45.077793 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:45.077695    2554 generic.go:358] "Generic (PLEG): container finished" podID="72404614-34dd-495d-9cbb-f0d0224422ac" containerID="510b4970090ea21333ebb98ba80c6114351c2f741d7dbd323ca199187ac78207" exitCode=0
May 20 13:59:45.077793 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:45.077767    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x" event={"ID":"72404614-34dd-495d-9cbb-f0d0224422ac","Type":"ContainerDied","Data":"510b4970090ea21333ebb98ba80c6114351c2f741d7dbd323ca199187ac78207"}
May 20 13:59:46.082732 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:46.082690    2554 generic.go:358] "Generic (PLEG): container finished" podID="72404614-34dd-495d-9cbb-f0d0224422ac" containerID="617567e75e4117c6dff22a016790048d75af0b55023a3ae3eaf6e7605125bc75" exitCode=0
May 20 13:59:46.083127 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:46.082745    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x" event={"ID":"72404614-34dd-495d-9cbb-f0d0224422ac","Type":"ContainerDied","Data":"617567e75e4117c6dff22a016790048d75af0b55023a3ae3eaf6e7605125bc75"}
May 20 13:59:47.215630 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.215604    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:47.267270 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.267236    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-util\") pod \"72404614-34dd-495d-9cbb-f0d0224422ac\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") "
May 20 13:59:47.267469 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.267308    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdn6d\" (UniqueName: \"kubernetes.io/projected/72404614-34dd-495d-9cbb-f0d0224422ac-kube-api-access-rdn6d\") pod \"72404614-34dd-495d-9cbb-f0d0224422ac\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") "
May 20 13:59:47.267469 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.267332    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-bundle\") pod \"72404614-34dd-495d-9cbb-f0d0224422ac\" (UID: \"72404614-34dd-495d-9cbb-f0d0224422ac\") "
May 20 13:59:47.268533 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.268501    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-bundle" (OuterVolumeSpecName: "bundle") pod "72404614-34dd-495d-9cbb-f0d0224422ac" (UID: "72404614-34dd-495d-9cbb-f0d0224422ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:59:47.269620 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.269599    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72404614-34dd-495d-9cbb-f0d0224422ac-kube-api-access-rdn6d" (OuterVolumeSpecName: "kube-api-access-rdn6d") pod "72404614-34dd-495d-9cbb-f0d0224422ac" (UID: "72404614-34dd-495d-9cbb-f0d0224422ac"). InnerVolumeSpecName "kube-api-access-rdn6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:59:47.273303 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.273275    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-util" (OuterVolumeSpecName: "util") pod "72404614-34dd-495d-9cbb-f0d0224422ac" (UID: "72404614-34dd-495d-9cbb-f0d0224422ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:59:47.369225 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.369120    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:59:47.369225 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.369164    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdn6d\" (UniqueName: \"kubernetes.io/projected/72404614-34dd-495d-9cbb-f0d0224422ac-kube-api-access-rdn6d\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:59:47.369225 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:47.369178    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72404614-34dd-495d-9cbb-f0d0224422ac-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 13:59:48.092815 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:48.092773    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x" event={"ID":"72404614-34dd-495d-9cbb-f0d0224422ac","Type":"ContainerDied","Data":"73c059ce28765e409b47cc60a636ea556f89455c67e448a54357a4d8ef594a83"}
May 20 13:59:48.092815 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:48.092810    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c059ce28765e409b47cc60a636ea556f89455c67e448a54357a4d8ef594a83"
May 20 13:59:48.092815 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:48.092811    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835pzt4x"
May 20 13:59:56.998053 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998020    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"]
May 20 13:59:56.998477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998345    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72404614-34dd-495d-9cbb-f0d0224422ac" containerName="pull"
May 20 13:59:56.998477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998357    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="72404614-34dd-495d-9cbb-f0d0224422ac" containerName="pull"
May 20 13:59:56.998477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998372    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72404614-34dd-495d-9cbb-f0d0224422ac" containerName="util"
May 20 13:59:56.998477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998378    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="72404614-34dd-495d-9cbb-f0d0224422ac" containerName="util"
May 20 13:59:56.998477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998383    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72404614-34dd-495d-9cbb-f0d0224422ac" containerName="extract"
May 20 13:59:56.998477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998389    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="72404614-34dd-495d-9cbb-f0d0224422ac" containerName="extract"
May 20 13:59:56.998477 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:56.998449    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="72404614-34dd-495d-9cbb-f0d0224422ac" containerName="extract"
May 20 13:59:57.001309 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.001288    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.004130 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.004104    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 13:59:57.005088 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.005063    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bm9kh\""
May 20 13:59:57.005190 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.005089    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 13:59:57.012991 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.012968    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"]
May 20 13:59:57.051529 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.051488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nrq\" (UniqueName: \"kubernetes.io/projected/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-kube-api-access-h9nrq\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.051710 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.051535    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.051710 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.051634    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.152693 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.152650    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nrq\" (UniqueName: \"kubernetes.io/projected/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-kube-api-access-h9nrq\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.152693 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.152697    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.152936 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.152735    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.153097 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.153080    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.153144 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.153127    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.194914 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.194882    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nrq\" (UniqueName: \"kubernetes.io/projected/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-kube-api-access-h9nrq\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.311308 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.311207    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 13:59:57.459502 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:57.459469    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"]
May 20 13:59:57.462804 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:59:57.462772    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc88e40_9399_4a01_bbbc_abffd9ffb85a.slice/crio-3c35a943f58d4efea608b1ad10b58e26bc8bf72659fc4b482b224ddf52ae9e38 WatchSource:0}: Error finding container 3c35a943f58d4efea608b1ad10b58e26bc8bf72659fc4b482b224ddf52ae9e38: Status 404 returned error can't find the container with id 3c35a943f58d4efea608b1ad10b58e26bc8bf72659fc4b482b224ddf52ae9e38
May 20 13:59:58.130319 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.130275    2554 generic.go:358] "Generic (PLEG): container finished" podID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerID="8344decc2f05e4877694be50c06934cff083c306e4a933855fc01d2c6a09c0fd" exitCode=0
May 20 13:59:58.130716 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.130365    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js" event={"ID":"5fc88e40-9399-4a01-bbbc-abffd9ffb85a","Type":"ContainerDied","Data":"8344decc2f05e4877694be50c06934cff083c306e4a933855fc01d2c6a09c0fd"}
May 20 13:59:58.130716 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.130408    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js" event={"ID":"5fc88e40-9399-4a01-bbbc-abffd9ffb85a","Type":"ContainerStarted","Data":"3c35a943f58d4efea608b1ad10b58e26bc8bf72659fc4b482b224ddf52ae9e38"}
May 20 13:59:58.257396 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.257359    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"]
May 20 13:59:58.259828 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.259809    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.263668 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.263644    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\""
May 20 13:59:58.264430 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.264413    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\""
May 20 13:59:58.264532 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.264420    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-vghfz\""
May 20 13:59:58.281373 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.281338    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"]
May 20 13:59:58.363897 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.363832    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46mh\" (UniqueName: \"kubernetes.io/projected/f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0-kube-api-access-c46mh\") pod \"servicemesh-operator3-55f49c5f94-54kcb\" (UID: \"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.364070 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.363982    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0-operator-config\") pod \"servicemesh-operator3-55f49c5f94-54kcb\" (UID: \"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.464979 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.464941    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0-operator-config\") pod \"servicemesh-operator3-55f49c5f94-54kcb\" (UID: \"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.465172 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.465022    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c46mh\" (UniqueName: \"kubernetes.io/projected/f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0-kube-api-access-c46mh\") pod \"servicemesh-operator3-55f49c5f94-54kcb\" (UID: \"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.467437 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.467405    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0-operator-config\") pod \"servicemesh-operator3-55f49c5f94-54kcb\" (UID: \"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.475397 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.475354    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46mh\" (UniqueName: \"kubernetes.io/projected/f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0-kube-api-access-c46mh\") pod \"servicemesh-operator3-55f49c5f94-54kcb\" (UID: \"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.569541 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.569486    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 13:59:58.708748 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:58.708663    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"]
May 20 13:59:58.711101 ip-10-0-141-251 kubenswrapper[2554]: W0520 13:59:58.711076    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bf4a1d_34fd_48dd_b4d6_cb759c6c79b0.slice/crio-fb3c14854cf777e18627bd0fcc1318de69e74603feee476c2ceae020320ad843 WatchSource:0}: Error finding container fb3c14854cf777e18627bd0fcc1318de69e74603feee476c2ceae020320ad843: Status 404 returned error can't find the container with id fb3c14854cf777e18627bd0fcc1318de69e74603feee476c2ceae020320ad843
May 20 13:59:59.135645 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:59.135616    2554 generic.go:358] "Generic (PLEG): container finished" podID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerID="708363f901f59aa4d5967e196ac395212ee1f0b7649053034caeefcd5c3f8a0f" exitCode=0
May 20 13:59:59.136038 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:59.135709    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js" event={"ID":"5fc88e40-9399-4a01-bbbc-abffd9ffb85a","Type":"ContainerDied","Data":"708363f901f59aa4d5967e196ac395212ee1f0b7649053034caeefcd5c3f8a0f"}
May 20 13:59:59.137175 ip-10-0-141-251 kubenswrapper[2554]: I0520 13:59:59.137148    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb" event={"ID":"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0","Type":"ContainerStarted","Data":"fb3c14854cf777e18627bd0fcc1318de69e74603feee476c2ceae020320ad843"}
May 20 14:00:00.141898 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:00.141863    2554 generic.go:358] "Generic (PLEG): container finished" podID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerID="fed52ff03ad4ea07466121cc0ca3858217e773c0d2b7e378e078e0e3d53e8c95" exitCode=0
May 20 14:00:00.142277 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:00.141941    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js" event={"ID":"5fc88e40-9399-4a01-bbbc-abffd9ffb85a","Type":"ContainerDied","Data":"fed52ff03ad4ea07466121cc0ca3858217e773c0d2b7e378e078e0e3d53e8c95"}
May 20 14:00:01.296418 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.296393    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 14:00:01.391241 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.391208    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9nrq\" (UniqueName: \"kubernetes.io/projected/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-kube-api-access-h9nrq\") pod \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") "
May 20 14:00:01.391410 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.391397    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-bundle\") pod \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") "
May 20 14:00:01.391453 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.391420    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-util\") pod \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\" (UID: \"5fc88e40-9399-4a01-bbbc-abffd9ffb85a\") "
May 20 14:00:01.392355 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.392327    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-bundle" (OuterVolumeSpecName: "bundle") pod "5fc88e40-9399-4a01-bbbc-abffd9ffb85a" (UID: "5fc88e40-9399-4a01-bbbc-abffd9ffb85a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:01.393462 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.393426    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-kube-api-access-h9nrq" (OuterVolumeSpecName: "kube-api-access-h9nrq") pod "5fc88e40-9399-4a01-bbbc-abffd9ffb85a" (UID: "5fc88e40-9399-4a01-bbbc-abffd9ffb85a"). InnerVolumeSpecName "kube-api-access-h9nrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:00:01.398213 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.398080    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-util" (OuterVolumeSpecName: "util") pod "5fc88e40-9399-4a01-bbbc-abffd9ffb85a" (UID: "5fc88e40-9399-4a01-bbbc-abffd9ffb85a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:01.492879 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.492827    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:01.492879 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.492879    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:01.492879 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:01.492889    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9nrq\" (UniqueName: \"kubernetes.io/projected/5fc88e40-9399-4a01-bbbc-abffd9ffb85a-kube-api-access-h9nrq\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:02.154523 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:02.154490    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js"
May 20 14:00:02.154707 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:02.154493    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azd2js" event={"ID":"5fc88e40-9399-4a01-bbbc-abffd9ffb85a","Type":"ContainerDied","Data":"3c35a943f58d4efea608b1ad10b58e26bc8bf72659fc4b482b224ddf52ae9e38"}
May 20 14:00:02.154707 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:02.154619    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c35a943f58d4efea608b1ad10b58e26bc8bf72659fc4b482b224ddf52ae9e38"
May 20 14:00:04.163558 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.163515    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb" event={"ID":"f7bf4a1d-34fd-48dd-b4d6-cb759c6c79b0","Type":"ContainerStarted","Data":"8d01a5ef56bc5f60fa07660f90b58e95861b4b13a4f3d984d2f9931ea413be8e"}
May 20 14:00:04.164050 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.163612    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 14:00:04.186346 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.186281    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb" podStartSLOduration=1.318987835 podStartE2EDuration="6.186258863s" podCreationTimestamp="2026-05-20 13:59:58 +0000 UTC" firstStartedPulling="2026-05-20 13:59:58.71364833 +0000 UTC m=+583.336373377" lastFinishedPulling="2026-05-20 14:00:03.580919355 +0000 UTC m=+588.203644405" observedRunningTime="2026-05-20 14:00:04.184784493 +0000 UTC m=+588.807509561" watchObservedRunningTime="2026-05-20 14:00:04.186258863 +0000 UTC m=+588.808983932"
May 20 14:00:04.629212 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629120    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"]
May 20 14:00:04.629525 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629507    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerName="pull"
May 20 14:00:04.629597 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629528    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerName="pull"
May 20 14:00:04.629597 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629540    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerName="extract"
May 20 14:00:04.629597 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629546    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerName="extract"
May 20 14:00:04.629597 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629567    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerName="util"
May 20 14:00:04.629597 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629573    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerName="util"
May 20 14:00:04.629819 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.629626    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="5fc88e40-9399-4a01-bbbc-abffd9ffb85a" containerName="extract"
May 20 14:00:04.632281 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.632263    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.634951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.634925    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
May 20 14:00:04.634951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.634956    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
May 20 14:00:04.635120 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.634926    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
May 20 14:00:04.635120 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.635000    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zcsgc\""
May 20 14:00:04.635309 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.635291    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 20 14:00:04.635467 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.635359    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 20 14:00:04.635467 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.635442    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
May 20 14:00:04.644894 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.644858    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"]
May 20 14:00:04.722927 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.722885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.723102 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.722938    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.723102 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.722976    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.723102 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.723001    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.723102 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.723073    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.723237 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.723112    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.723237 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.723144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxb5v\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-kube-api-access-lxb5v\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.824403 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.824361    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.824403 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.824403    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.824675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.824436    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lxb5v\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-kube-api-access-lxb5v\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.824675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.824461    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.824675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.824482    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.824675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.824510    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.824675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.824533    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.825229 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.825202    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.827423 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.827355    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-kubeconfig\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.827423 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.827391    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.827594 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.827428    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-local-certs\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.827594 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.827509    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-cacerts\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.835318 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.835260    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-token\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.835940 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.835679    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxb5v\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-kube-api-access-lxb5v\") pod \"istiod-openshift-gateway-767cf5cf55-zzv8x\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") " pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:04.941959 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:04.941916    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:05.090497 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:05.090470    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"]
May 20 14:00:05.092887 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:05.092827    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23ad75c_59c6_4a8b_a472_1bc320e04cf9.slice/crio-3e3866638038d8af07aaef88c44a139c298d53fdc12998a153c3e893fd8335b0 WatchSource:0}: Error finding container 3e3866638038d8af07aaef88c44a139c298d53fdc12998a153c3e893fd8335b0: Status 404 returned error can't find the container with id 3e3866638038d8af07aaef88c44a139c298d53fdc12998a153c3e893fd8335b0
May 20 14:00:05.168576 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:05.168534    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x" event={"ID":"b23ad75c-59c6-4a8b-a472-1bc320e04cf9","Type":"ContainerStarted","Data":"3e3866638038d8af07aaef88c44a139c298d53fdc12998a153c3e893fd8335b0"}
May 20 14:00:08.840962 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.840926    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-774b44c455-ffzwg"]
May 20 14:00:08.844180 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.844151    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:08.861800 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.861765    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774b44c455-ffzwg"]
May 20 14:00:08.964231 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.964191    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-service-ca\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:08.964419 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.964257    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lbl6\" (UniqueName: \"kubernetes.io/projected/24e8b27d-13fa-4a0f-9220-63e048705b7d-kube-api-access-8lbl6\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:08.964419 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.964292    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-serving-cert\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:08.964419 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.964323    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-config\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:08.964419 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.964387    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-trusted-ca-bundle\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:08.964591 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.964445    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-oauth-config\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:08.964591 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:08.964475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-oauth-serving-cert\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.065254 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.065214    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-oauth-config\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.065430 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.065265    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-oauth-serving-cert\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.065430 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.065331    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-service-ca\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.065430 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.065363    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8lbl6\" (UniqueName: \"kubernetes.io/projected/24e8b27d-13fa-4a0f-9220-63e048705b7d-kube-api-access-8lbl6\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.065430 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.065390    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-serving-cert\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.065430 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.065418    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-config\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.065727 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.065460    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-trusted-ca-bundle\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.066178 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.066141    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-service-ca\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.066296 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.066141    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-oauth-serving-cert\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.066376 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.066347    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-config\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.066500 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.066351    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24e8b27d-13fa-4a0f-9220-63e048705b7d-trusted-ca-bundle\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.068202 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.068180    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-oauth-config\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.068423 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.068399    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/24e8b27d-13fa-4a0f-9220-63e048705b7d-console-serving-cert\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.074684 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.074659    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lbl6\" (UniqueName: \"kubernetes.io/projected/24e8b27d-13fa-4a0f-9220-63e048705b7d-kube-api-access-8lbl6\") pod \"console-774b44c455-ffzwg\" (UID: \"24e8b27d-13fa-4a0f-9220-63e048705b7d\") " pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.154775 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.154725    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:09.311609 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:09.311583    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774b44c455-ffzwg"]
May 20 14:00:09.313937 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:09.313897    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e8b27d_13fa_4a0f_9220_63e048705b7d.slice/crio-47cabb2be52152843dc22ee7fee4268759c81baa2c7660f6306952204f4cdbfc WatchSource:0}: Error finding container 47cabb2be52152843dc22ee7fee4268759c81baa2c7660f6306952204f4cdbfc: Status 404 returned error can't find the container with id 47cabb2be52152843dc22ee7fee4268759c81baa2c7660f6306952204f4cdbfc
May 20 14:00:10.188517 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:10.188466    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774b44c455-ffzwg" event={"ID":"24e8b27d-13fa-4a0f-9220-63e048705b7d","Type":"ContainerStarted","Data":"4c50a2450662697bb5c9924fc91d37f7c8be39ad0792bc594e198586a9d14e3b"}
May 20 14:00:10.188517 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:10.188512    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774b44c455-ffzwg" event={"ID":"24e8b27d-13fa-4a0f-9220-63e048705b7d","Type":"ContainerStarted","Data":"47cabb2be52152843dc22ee7fee4268759c81baa2c7660f6306952204f4cdbfc"}
May 20 14:00:10.212743 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:10.212683    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-774b44c455-ffzwg" podStartSLOduration=2.212663706 podStartE2EDuration="2.212663706s" podCreationTimestamp="2026-05-20 14:00:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:00:10.211530726 +0000 UTC m=+594.834255800" watchObservedRunningTime="2026-05-20 14:00:10.212663706 +0000 UTC m=+594.835388776"
May 20 14:00:10.731289 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:10.731244    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:00:10.731369 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:10.731328    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:00:11.196360 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:11.196040    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x" event={"ID":"b23ad75c-59c6-4a8b-a472-1bc320e04cf9","Type":"ContainerStarted","Data":"3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b"}
May 20 14:00:11.196360 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:11.196321    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:11.198252 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:11.198224    2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-767cf5cf55-zzv8x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
May 20 14:00:11.198386 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:11.198277    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x" podUID="b23ad75c-59c6-4a8b-a472-1bc320e04cf9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:00:11.221084 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:11.221016    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x" podStartSLOduration=1.584943897 podStartE2EDuration="7.220996919s" podCreationTimestamp="2026-05-20 14:00:04 +0000 UTC" firstStartedPulling="2026-05-20 14:00:05.094957613 +0000 UTC m=+589.717682660" lastFinishedPulling="2026-05-20 14:00:10.731010623 +0000 UTC m=+595.353735682" observedRunningTime="2026-05-20 14:00:11.218224427 +0000 UTC m=+595.840949495" watchObservedRunningTime="2026-05-20 14:00:11.220996919 +0000 UTC m=+595.843721989"
May 20 14:00:12.200100 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.200063    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:00:12.554742 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.554654    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"]
May 20 14:00:12.558222 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.558206    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.560946 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.560919    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-wt2d6\""
May 20 14:00:12.573892 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.573862    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"]
May 20 14:00:12.701746 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701707    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-workload-socket\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.701746 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701745    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-envoy\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.701979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701780    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-credential-socket\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.701979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701801    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.701979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701829    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-token\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.701979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701869    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-workload-certs\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.701979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701925    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhql\" (UniqueName: \"kubernetes.io/projected/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-kube-api-access-5hhql\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.701979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.701960    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.702158 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.702012    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-data\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803257 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803216    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-data\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803446 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803275    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-workload-socket\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803446 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803295    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-envoy\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803446 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803412    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-credential-socket\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803582 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803462    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803582 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803507    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-token\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803582 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803535    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-workload-certs\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803582 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803577    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhql\" (UniqueName: \"kubernetes.io/projected/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-kube-api-access-5hhql\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803769 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803618    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803769 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803754    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-data\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.803890 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.803787    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-workload-socket\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.804173 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.804142    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-workload-certs\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.804291 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.804228    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-credential-socket\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.804291 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.804266    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.805677 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.805624    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-envoy\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.806078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.806053    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.812023 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.811998    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-istio-token\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.812104 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.812069    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhql\" (UniqueName: \"kubernetes.io/projected/f3240ff9-7b7b-4cbe-a8de-841b7f425c80-kube-api-access-5hhql\") pod \"openshift-ai-inference-openshift-default-84b84d7ccf-g66f7\" (UID: \"f3240ff9-7b7b-4cbe-a8de-841b7f425c80\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:12.870073 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:12.870026    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:13.015600 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:13.015563    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"]
May 20 14:00:13.017515 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:13.017477    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3240ff9_7b7b_4cbe_a8de_841b7f425c80.slice/crio-562a911af3f58e0a396d43d8d902e8ba2de0d8d8c2b0f000dd5902d9825a2d85 WatchSource:0}: Error finding container 562a911af3f58e0a396d43d8d902e8ba2de0d8d8c2b0f000dd5902d9825a2d85: Status 404 returned error can't find the container with id 562a911af3f58e0a396d43d8d902e8ba2de0d8d8c2b0f000dd5902d9825a2d85
May 20 14:00:13.205139 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:13.205095    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7" event={"ID":"f3240ff9-7b7b-4cbe-a8de-841b7f425c80","Type":"ContainerStarted","Data":"562a911af3f58e0a396d43d8d902e8ba2de0d8d8c2b0f000dd5902d9825a2d85"}
May 20 14:00:15.171972 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:15.171932    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-54kcb"
May 20 14:00:15.908627 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:15.908598    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:00:15.908804 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:15.908682    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:00:15.948428 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:15.948391    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:00:15.948534 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:15.948461    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:00:15.948534 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:15.948490    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:00:16.225854 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:16.225800    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7" event={"ID":"f3240ff9-7b7b-4cbe-a8de-841b7f425c80","Type":"ContainerStarted","Data":"cb5ba2e013bd8c201f1bfd88e9813aaba0a0b2c135be5c468831f94ea9f39cda"}
May 20 14:00:16.294684 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:16.294611    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7" podStartSLOduration=1.3659380620000001 podStartE2EDuration="4.294592978s" podCreationTimestamp="2026-05-20 14:00:12 +0000 UTC" firstStartedPulling="2026-05-20 14:00:13.019451852 +0000 UTC m=+597.642176901" lastFinishedPulling="2026-05-20 14:00:15.948106772 +0000 UTC m=+600.570831817" observedRunningTime="2026-05-20 14:00:16.291699231 +0000 UTC m=+600.914424300" watchObservedRunningTime="2026-05-20 14:00:16.294592978 +0000 UTC m=+600.917318050"
May 20 14:00:16.870670 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:16.870614    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:17.875986 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:17.875952    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:18.233064 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:18.233031    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:18.233961 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:18.233941    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-84b84d7ccf-g66f7"
May 20 14:00:19.155231 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:19.155194    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:19.155586 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:19.155245    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:19.160118 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:19.160090    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:19.240273 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:19.240241    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774b44c455-ffzwg"
May 20 14:00:19.333524 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:19.333491    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7457dcf47b-gxczx"]
May 20 14:00:23.399152 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.399105    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"]
May 20 14:00:23.404955 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.404926    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.409683 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.409659    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 14:00:23.409683 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.409659    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 14:00:23.417032 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.417011    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bm9kh\""
May 20 14:00:23.422382 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.422352    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"]
May 20 14:00:23.504079 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.504038    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.504079 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.504076    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.504297 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.504124    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsjb\" (UniqueName: \"kubernetes.io/projected/f76c9bf3-7388-4aa2-886e-e565b413da58-kube-api-access-zhsjb\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.515642 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.515608    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"]
May 20 14:00:23.519086 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.519069    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.527864 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.527807    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"]
May 20 14:00:23.582362 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.582323    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"]
May 20 14:00:23.586005 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.585984    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.594557 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.594529    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"]
May 20 14:00:23.605149 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605119    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.605326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605155    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.605326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605177    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.605326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605214    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsjb\" (UniqueName: \"kubernetes.io/projected/f76c9bf3-7388-4aa2-886e-e565b413da58-kube-api-access-zhsjb\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.605326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.605326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605271    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.605326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605296    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6db\" (UniqueName: \"kubernetes.io/projected/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-kube-api-access-8c6db\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.605326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.605676 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwkr\" (UniqueName: \"kubernetes.io/projected/3e6d3169-6c45-440a-8ec6-fe5232d285f8-kube-api-access-vnwkr\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.605676 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605536    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.605676 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.605600    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.613517 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.613492    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsjb\" (UniqueName: \"kubernetes.io/projected/f76c9bf3-7388-4aa2-886e-e565b413da58-kube-api-access-zhsjb\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.685455 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.685420    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"]
May 20 14:00:23.689209 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.689187    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.697252 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.697225    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"]
May 20 14:00:23.706228 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706186    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.706389 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706235    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwkr\" (UniqueName: \"kubernetes.io/projected/3e6d3169-6c45-440a-8ec6-fe5232d285f8-kube-api-access-vnwkr\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.706389 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706307    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.706389 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706362    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slsj8\" (UniqueName: \"kubernetes.io/projected/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-kube-api-access-slsj8\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.706571 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706411    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.706571 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706436    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.706571 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706480    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.706571 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706526    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.706571 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706551    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6db\" (UniqueName: \"kubernetes.io/projected/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-kube-api-access-8c6db\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.706855 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706664    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.707018 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.706993    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.707183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.707161    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.707395 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.707373    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.714825 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.714798    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:23.715818 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.715429    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6db\" (UniqueName: \"kubernetes.io/projected/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-kube-api-access-8c6db\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.715907 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.715875    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwkr\" (UniqueName: \"kubernetes.io/projected/3e6d3169-6c45-440a-8ec6-fe5232d285f8-kube-api-access-vnwkr\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.807733 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.807693    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.807907 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.807777    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-slsj8\" (UniqueName: \"kubernetes.io/projected/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-kube-api-access-slsj8\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.807907 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.807827    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.808205 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.808179    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.808249 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.808217    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.817893 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.817820    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slsj8\" (UniqueName: \"kubernetes.io/projected/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-kube-api-access-slsj8\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:23.828695 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.828666    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:23.850458 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.850424    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"]
May 20 14:00:23.851767 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:23.851739    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76c9bf3_7388_4aa2_886e_e565b413da58.slice/crio-4d05a9cc1593cd4c602059ce14cc04c5f81ab5d0bd2349d759e871ece9ea9934 WatchSource:0}: Error finding container 4d05a9cc1593cd4c602059ce14cc04c5f81ab5d0bd2349d759e871ece9ea9934: Status 404 returned error can't find the container with id 4d05a9cc1593cd4c602059ce14cc04c5f81ab5d0bd2349d759e871ece9ea9934
May 20 14:00:23.896169 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.896125    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:23.977088 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:23.976918    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"]
May 20 14:00:24.000293 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.000258    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:24.026345 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:24.026299    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b886aa_b19e_4ac8_a554_8fbfa11fc61e.slice/crio-afe9abb44a588677fd984ff6e66be3b863287e5c8332390473189918b29d8ad0 WatchSource:0}: Error finding container afe9abb44a588677fd984ff6e66be3b863287e5c8332390473189918b29d8ad0: Status 404 returned error can't find the container with id afe9abb44a588677fd984ff6e66be3b863287e5c8332390473189918b29d8ad0
May 20 14:00:24.047637 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.047595    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"]
May 20 14:00:24.152675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.152647    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"]
May 20 14:00:24.211090 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:24.211004    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b15b138_f9ae_4ca1_851d_4f087eb2c4f8.slice/crio-b8c6b4ee4f7c3236e48b84352bab490b6a31d6e10741b101ec8f432f1d0a76b6 WatchSource:0}: Error finding container b8c6b4ee4f7c3236e48b84352bab490b6a31d6e10741b101ec8f432f1d0a76b6: Status 404 returned error can't find the container with id b8c6b4ee4f7c3236e48b84352bab490b6a31d6e10741b101ec8f432f1d0a76b6
May 20 14:00:24.255890 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.255861    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7" event={"ID":"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8","Type":"ContainerStarted","Data":"b8c6b4ee4f7c3236e48b84352bab490b6a31d6e10741b101ec8f432f1d0a76b6"}
May 20 14:00:24.257159 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.257137    2554 generic.go:358] "Generic (PLEG): container finished" podID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerID="9d14f1ed471f334acaac3de1776213b688d4a16c2d24412737dea77f53faefb9" exitCode=0
May 20 14:00:24.257263 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.257219    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl" event={"ID":"f76c9bf3-7388-4aa2-886e-e565b413da58","Type":"ContainerDied","Data":"9d14f1ed471f334acaac3de1776213b688d4a16c2d24412737dea77f53faefb9"}
May 20 14:00:24.257263 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.257256    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl" event={"ID":"f76c9bf3-7388-4aa2-886e-e565b413da58","Type":"ContainerStarted","Data":"4d05a9cc1593cd4c602059ce14cc04c5f81ab5d0bd2349d759e871ece9ea9934"}
May 20 14:00:24.258594 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.258487    2554 generic.go:358] "Generic (PLEG): container finished" podID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerID="9d38f2158af6d3f4879f14cb541c7936d91f08aadc66473b805ba862eafe301b" exitCode=0
May 20 14:00:24.258594 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.258563    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf" event={"ID":"3e6d3169-6c45-440a-8ec6-fe5232d285f8","Type":"ContainerDied","Data":"9d38f2158af6d3f4879f14cb541c7936d91f08aadc66473b805ba862eafe301b"}
May 20 14:00:24.258594 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.258590    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf" event={"ID":"3e6d3169-6c45-440a-8ec6-fe5232d285f8","Type":"ContainerStarted","Data":"a2845f7846d3c6333bbba3502938435a13439d16b00919a4f728c91595d1788f"}
May 20 14:00:24.259851 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.259818    2554 generic.go:358] "Generic (PLEG): container finished" podID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerID="b68bc109b9327e2cb202b55344585d93fe08c2669e9d279e8152361e116ef23c" exitCode=0
May 20 14:00:24.259933 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.259897    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp" event={"ID":"37b886aa-b19e-4ac8-a554-8fbfa11fc61e","Type":"ContainerDied","Data":"b68bc109b9327e2cb202b55344585d93fe08c2669e9d279e8152361e116ef23c"}
May 20 14:00:24.259933 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:24.259929    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp" event={"ID":"37b886aa-b19e-4ac8-a554-8fbfa11fc61e","Type":"ContainerStarted","Data":"afe9abb44a588677fd984ff6e66be3b863287e5c8332390473189918b29d8ad0"}
May 20 14:00:25.265300 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:25.265258    2554 generic.go:358] "Generic (PLEG): container finished" podID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerID="bca36700036bc6b5ec4a38061655b4e5abf3fe17dd2c2f3f670cfec1b48aab8a" exitCode=0
May 20 14:00:25.265702 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:25.265349    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7" event={"ID":"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8","Type":"ContainerDied","Data":"bca36700036bc6b5ec4a38061655b4e5abf3fe17dd2c2f3f670cfec1b48aab8a"}
May 20 14:00:26.272043 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:26.272009    2554 generic.go:358] "Generic (PLEG): container finished" podID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerID="2e5fcc880224cf4f07f0eec959e72a6af0f509fefdee18aa3aaa65a84aff9ff7" exitCode=0
May 20 14:00:26.272467 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:26.272095    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf" event={"ID":"3e6d3169-6c45-440a-8ec6-fe5232d285f8","Type":"ContainerDied","Data":"2e5fcc880224cf4f07f0eec959e72a6af0f509fefdee18aa3aaa65a84aff9ff7"}
May 20 14:00:26.273831 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:26.273729    2554 generic.go:358] "Generic (PLEG): container finished" podID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerID="9b7a9c9c81641f7ba68f97deee46f0c5a46528a040dc1805cd5759d2eed10ec7" exitCode=0
May 20 14:00:26.273831 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:26.273812    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp" event={"ID":"37b886aa-b19e-4ac8-a554-8fbfa11fc61e","Type":"ContainerDied","Data":"9b7a9c9c81641f7ba68f97deee46f0c5a46528a040dc1805cd5759d2eed10ec7"}
May 20 14:00:26.275560 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:26.275538    2554 generic.go:358] "Generic (PLEG): container finished" podID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerID="1075498bfe2ffe9fe4c8c8b0bcd9af927b8b67be138c694519ea93ddf3a64d3a" exitCode=0
May 20 14:00:26.275637 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:26.275592    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl" event={"ID":"f76c9bf3-7388-4aa2-886e-e565b413da58","Type":"ContainerDied","Data":"1075498bfe2ffe9fe4c8c8b0bcd9af927b8b67be138c694519ea93ddf3a64d3a"}
May 20 14:00:27.280999 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.280958    2554 generic.go:358] "Generic (PLEG): container finished" podID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerID="d784c5329443d20ffd13a69a9d305c604f8dde109f32cd39a57a87fa6a058656" exitCode=0
May 20 14:00:27.281471 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.281050    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7" event={"ID":"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8","Type":"ContainerDied","Data":"d784c5329443d20ffd13a69a9d305c604f8dde109f32cd39a57a87fa6a058656"}
May 20 14:00:27.282979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.282959    2554 generic.go:358] "Generic (PLEG): container finished" podID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerID="030f36f74760a7e5c3ec987a9e2e3c0616b7e2576beb38da80ff866bcd22928a" exitCode=0
May 20 14:00:27.283062 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.283035    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl" event={"ID":"f76c9bf3-7388-4aa2-886e-e565b413da58","Type":"ContainerDied","Data":"030f36f74760a7e5c3ec987a9e2e3c0616b7e2576beb38da80ff866bcd22928a"}
May 20 14:00:27.284828 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.284807    2554 generic.go:358] "Generic (PLEG): container finished" podID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerID="5153f5c25d90e1b74104776ea113a567c547fdcd7a51a076e58f1d06649d6d45" exitCode=0
May 20 14:00:27.284965 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.284863    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf" event={"ID":"3e6d3169-6c45-440a-8ec6-fe5232d285f8","Type":"ContainerDied","Data":"5153f5c25d90e1b74104776ea113a567c547fdcd7a51a076e58f1d06649d6d45"}
May 20 14:00:27.286716 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.286694    2554 generic.go:358] "Generic (PLEG): container finished" podID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerID="502a1c3a0d26cce40ae5091ded670889fede256a8717181e1a1cdeede82851a0" exitCode=0
May 20 14:00:27.286803 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:27.286768    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp" event={"ID":"37b886aa-b19e-4ac8-a554-8fbfa11fc61e","Type":"ContainerDied","Data":"502a1c3a0d26cce40ae5091ded670889fede256a8717181e1a1cdeede82851a0"}
May 20 14:00:28.293081 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.293042    2554 generic.go:358] "Generic (PLEG): container finished" podID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerID="2a926c9fe04d4cd97185812967658a87b7128ce61c6ecf3a14ac2f1243975d8d" exitCode=0
May 20 14:00:28.293453 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.293124    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7" event={"ID":"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8","Type":"ContainerDied","Data":"2a926c9fe04d4cd97185812967658a87b7128ce61c6ecf3a14ac2f1243975d8d"}
May 20 14:00:28.464579 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.464548    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:28.476986 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.476964    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:28.480311 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.480291    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:28.545592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545500    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-util\") pod \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") "
May 20 14:00:28.545592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545549    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c6db\" (UniqueName: \"kubernetes.io/projected/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-kube-api-access-8c6db\") pod \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") "
May 20 14:00:28.545592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545587    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhsjb\" (UniqueName: \"kubernetes.io/projected/f76c9bf3-7388-4aa2-886e-e565b413da58-kube-api-access-zhsjb\") pod \"f76c9bf3-7388-4aa2-886e-e565b413da58\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") "
May 20 14:00:28.545897 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545637    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-util\") pod \"f76c9bf3-7388-4aa2-886e-e565b413da58\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") "
May 20 14:00:28.545897 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545659    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-bundle\") pod \"f76c9bf3-7388-4aa2-886e-e565b413da58\" (UID: \"f76c9bf3-7388-4aa2-886e-e565b413da58\") "
May 20 14:00:28.545897 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545795    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-util\") pod \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") "
May 20 14:00:28.545897 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545870    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-bundle\") pod \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\" (UID: \"37b886aa-b19e-4ac8-a554-8fbfa11fc61e\") "
May 20 14:00:28.546098 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545926    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-bundle\") pod \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") "
May 20 14:00:28.546098 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.545958    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnwkr\" (UniqueName: \"kubernetes.io/projected/3e6d3169-6c45-440a-8ec6-fe5232d285f8-kube-api-access-vnwkr\") pod \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\" (UID: \"3e6d3169-6c45-440a-8ec6-fe5232d285f8\") "
May 20 14:00:28.546462 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.546291    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-bundle" (OuterVolumeSpecName: "bundle") pod "f76c9bf3-7388-4aa2-886e-e565b413da58" (UID: "f76c9bf3-7388-4aa2-886e-e565b413da58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:28.546797 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.546764    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-bundle" (OuterVolumeSpecName: "bundle") pod "3e6d3169-6c45-440a-8ec6-fe5232d285f8" (UID: "3e6d3169-6c45-440a-8ec6-fe5232d285f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:28.547072 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.547048    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-bundle" (OuterVolumeSpecName: "bundle") pod "37b886aa-b19e-4ac8-a554-8fbfa11fc61e" (UID: "37b886aa-b19e-4ac8-a554-8fbfa11fc61e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:28.548302 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.548267    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-kube-api-access-8c6db" (OuterVolumeSpecName: "kube-api-access-8c6db") pod "37b886aa-b19e-4ac8-a554-8fbfa11fc61e" (UID: "37b886aa-b19e-4ac8-a554-8fbfa11fc61e"). InnerVolumeSpecName "kube-api-access-8c6db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:00:28.548408 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.548344    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76c9bf3-7388-4aa2-886e-e565b413da58-kube-api-access-zhsjb" (OuterVolumeSpecName: "kube-api-access-zhsjb") pod "f76c9bf3-7388-4aa2-886e-e565b413da58" (UID: "f76c9bf3-7388-4aa2-886e-e565b413da58"). InnerVolumeSpecName "kube-api-access-zhsjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:00:28.548408 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.548381    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6d3169-6c45-440a-8ec6-fe5232d285f8-kube-api-access-vnwkr" (OuterVolumeSpecName: "kube-api-access-vnwkr") pod "3e6d3169-6c45-440a-8ec6-fe5232d285f8" (UID: "3e6d3169-6c45-440a-8ec6-fe5232d285f8"). InnerVolumeSpecName "kube-api-access-vnwkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:00:28.552053 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.552020    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-util" (OuterVolumeSpecName: "util") pod "37b886aa-b19e-4ac8-a554-8fbfa11fc61e" (UID: "37b886aa-b19e-4ac8-a554-8fbfa11fc61e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:28.552402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.552378    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-util" (OuterVolumeSpecName: "util") pod "3e6d3169-6c45-440a-8ec6-fe5232d285f8" (UID: "3e6d3169-6c45-440a-8ec6-fe5232d285f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:28.552607 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.552585    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-util" (OuterVolumeSpecName: "util") pod "f76c9bf3-7388-4aa2-886e-e565b413da58" (UID: "f76c9bf3-7388-4aa2-886e-e565b413da58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:28.647557 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647516    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647557 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647550    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwkr\" (UniqueName: \"kubernetes.io/projected/3e6d3169-6c45-440a-8ec6-fe5232d285f8-kube-api-access-vnwkr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647557 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647561    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647571    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6db\" (UniqueName: \"kubernetes.io/projected/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-kube-api-access-8c6db\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647582    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsjb\" (UniqueName: \"kubernetes.io/projected/f76c9bf3-7388-4aa2-886e-e565b413da58-kube-api-access-zhsjb\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647591    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647600    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f76c9bf3-7388-4aa2-886e-e565b413da58-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647607    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3e6d3169-6c45-440a-8ec6-fe5232d285f8-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:28.647786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:28.647615    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37b886aa-b19e-4ac8-a554-8fbfa11fc61e-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:29.299671 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.299638    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl"
May 20 14:00:29.300150 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.299629    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec885sstl" event={"ID":"f76c9bf3-7388-4aa2-886e-e565b413da58","Type":"ContainerDied","Data":"4d05a9cc1593cd4c602059ce14cc04c5f81ab5d0bd2349d759e871ece9ea9934"}
May 20 14:00:29.300150 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.299757    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d05a9cc1593cd4c602059ce14cc04c5f81ab5d0bd2349d759e871ece9ea9934"
May 20 14:00:29.301362 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.301343    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf"
May 20 14:00:29.301463 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.301356    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b8gwrf" event={"ID":"3e6d3169-6c45-440a-8ec6-fe5232d285f8","Type":"ContainerDied","Data":"a2845f7846d3c6333bbba3502938435a13439d16b00919a4f728c91595d1788f"}
May 20 14:00:29.301463 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.301383    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2845f7846d3c6333bbba3502938435a13439d16b00919a4f728c91595d1788f"
May 20 14:00:29.303004 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.302985    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp"
May 20 14:00:29.303098 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.302983    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e785zkrp" event={"ID":"37b886aa-b19e-4ac8-a554-8fbfa11fc61e","Type":"ContainerDied","Data":"afe9abb44a588677fd984ff6e66be3b863287e5c8332390473189918b29d8ad0"}
May 20 14:00:29.303098 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.303030    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afe9abb44a588677fd984ff6e66be3b863287e5c8332390473189918b29d8ad0"
May 20 14:00:29.423445 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.423420    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:29.454683 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.454656    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-util\") pod \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") "
May 20 14:00:29.454867 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.454710    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-bundle\") pod \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") "
May 20 14:00:29.454867 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.454753    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slsj8\" (UniqueName: \"kubernetes.io/projected/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-kube-api-access-slsj8\") pod \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\" (UID: \"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8\") "
May 20 14:00:29.455262 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.455234    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-bundle" (OuterVolumeSpecName: "bundle") pod "2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" (UID: "2b15b138-f9ae-4ca1-851d-4f087eb2c4f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:29.456858 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.456809    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-kube-api-access-slsj8" (OuterVolumeSpecName: "kube-api-access-slsj8") pod "2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" (UID: "2b15b138-f9ae-4ca1-851d-4f087eb2c4f8"). InnerVolumeSpecName "kube-api-access-slsj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:00:29.461376 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.461345    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-util" (OuterVolumeSpecName: "util") pod "2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" (UID: "2b15b138-f9ae-4ca1-851d-4f087eb2c4f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:29.555760 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.555676    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:29.555760 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.555706    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slsj8\" (UniqueName: \"kubernetes.io/projected/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-kube-api-access-slsj8\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:29.555760 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:29.555715    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b15b138-f9ae-4ca1-851d-4f087eb2c4f8-util\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:30.308271 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:30.308240    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7"
May 20 14:00:30.308723 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:30.308242    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30tm4k7" event={"ID":"2b15b138-f9ae-4ca1-851d-4f087eb2c4f8","Type":"ContainerDied","Data":"b8c6b4ee4f7c3236e48b84352bab490b6a31d6e10741b101ec8f432f1d0a76b6"}
May 20 14:00:30.308723 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:30.308348    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c6b4ee4f7c3236e48b84352bab490b6a31d6e10741b101ec8f432f1d0a76b6"
May 20 14:00:35.950684 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.950599    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"]
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.950977    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.950989    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.950996    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951001    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951010    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951016    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951024    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951030    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951037    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951070    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951078    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951083    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951090    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951095    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951105    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951110    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951116    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951120    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951127    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951132    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerName="pull"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951138    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951144    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951153    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951158    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerName="util"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951213    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="3e6d3169-6c45-440a-8ec6-fe5232d285f8" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951223    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="37b886aa-b19e-4ac8-a554-8fbfa11fc61e" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951230    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b15b138-f9ae-4ca1-851d-4f087eb2c4f8" containerName="extract"
May 20 14:00:35.953040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.951237    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="f76c9bf3-7388-4aa2-886e-e565b413da58" containerName="extract"
May 20 14:00:35.954096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.954080    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:35.956697 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.956677    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l69kl\""
May 20 14:00:35.956827 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.956680    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 20 14:00:35.956827 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.956711    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 20 14:00:35.966381 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:35.966357    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"]
May 20 14:00:36.013322 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.013281    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab24b37-b09d-4300-945b-04bfa02377f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-z2bt4\" (UID: \"dab24b37-b09d-4300-945b-04bfa02377f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:36.013489 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.013336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dw49\" (UniqueName: \"kubernetes.io/projected/dab24b37-b09d-4300-945b-04bfa02377f1-kube-api-access-6dw49\") pod \"kuadrant-operator-controller-manager-57dd85bb4-z2bt4\" (UID: \"dab24b37-b09d-4300-945b-04bfa02377f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:36.114317 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.114283    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab24b37-b09d-4300-945b-04bfa02377f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-z2bt4\" (UID: \"dab24b37-b09d-4300-945b-04bfa02377f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:36.114484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.114341    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6dw49\" (UniqueName: \"kubernetes.io/projected/dab24b37-b09d-4300-945b-04bfa02377f1-kube-api-access-6dw49\") pod \"kuadrant-operator-controller-manager-57dd85bb4-z2bt4\" (UID: \"dab24b37-b09d-4300-945b-04bfa02377f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:36.114665 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.114636    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dab24b37-b09d-4300-945b-04bfa02377f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-z2bt4\" (UID: \"dab24b37-b09d-4300-945b-04bfa02377f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:36.131911 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.131881    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dw49\" (UniqueName: \"kubernetes.io/projected/dab24b37-b09d-4300-945b-04bfa02377f1-kube-api-access-6dw49\") pod \"kuadrant-operator-controller-manager-57dd85bb4-z2bt4\" (UID: \"dab24b37-b09d-4300-945b-04bfa02377f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:36.263815 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.263731    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:36.405818 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:36.405787    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"]
May 20 14:00:36.407564 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:36.407534    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab24b37_b09d_4300_945b_04bfa02377f1.slice/crio-a2a207d4268eb2dfb16c8c2a17e9a8646ee6e01f4c6d2984ab121dd4b6b0bb6a WatchSource:0}: Error finding container a2a207d4268eb2dfb16c8c2a17e9a8646ee6e01f4c6d2984ab121dd4b6b0bb6a: Status 404 returned error can't find the container with id a2a207d4268eb2dfb16c8c2a17e9a8646ee6e01f4c6d2984ab121dd4b6b0bb6a
May 20 14:00:37.337620 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:37.337575    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4" event={"ID":"dab24b37-b09d-4300-945b-04bfa02377f1","Type":"ContainerStarted","Data":"a2a207d4268eb2dfb16c8c2a17e9a8646ee6e01f4c6d2984ab121dd4b6b0bb6a"}
May 20 14:00:41.356264 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:41.356171    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4" event={"ID":"dab24b37-b09d-4300-945b-04bfa02377f1","Type":"ContainerStarted","Data":"ff3bd6e32c7d63296fecca01ddb598c566f52add2fd8efd349df0ca3b95b3a56"}
May 20 14:00:41.356629 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:41.356304    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:41.409963 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:41.409900    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4" podStartSLOduration=1.8191622509999998 podStartE2EDuration="6.409882764s" podCreationTimestamp="2026-05-20 14:00:35 +0000 UTC" firstStartedPulling="2026-05-20 14:00:36.410044016 +0000 UTC m=+621.032769077" lastFinishedPulling="2026-05-20 14:00:41.000764525 +0000 UTC m=+625.623489590" observedRunningTime="2026-05-20 14:00:41.408303426 +0000 UTC m=+626.031028494" watchObservedRunningTime="2026-05-20 14:00:41.409882764 +0000 UTC m=+626.032607831"
May 20 14:00:41.988037 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:41.987999    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"]
May 20 14:00:41.990466 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:41.990448    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"
May 20 14:00:41.995545 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:41.995517    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xgkkh\""
May 20 14:00:42.019446 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:42.019404    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"]
May 20 14:00:42.069892 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:42.069827    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbwb\" (UniqueName: \"kubernetes.io/projected/86e298cc-dd62-489e-9cb2-180bb03d395e-kube-api-access-wrbwb\") pod \"limitador-operator-controller-manager-c7fb4c8d5-qqh27\" (UID: \"86e298cc-dd62-489e-9cb2-180bb03d395e\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"
May 20 14:00:42.171246 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:42.171203    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbwb\" (UniqueName: \"kubernetes.io/projected/86e298cc-dd62-489e-9cb2-180bb03d395e-kube-api-access-wrbwb\") pod \"limitador-operator-controller-manager-c7fb4c8d5-qqh27\" (UID: \"86e298cc-dd62-489e-9cb2-180bb03d395e\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"
May 20 14:00:42.192410 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:42.192377    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbwb\" (UniqueName: \"kubernetes.io/projected/86e298cc-dd62-489e-9cb2-180bb03d395e-kube-api-access-wrbwb\") pod \"limitador-operator-controller-manager-c7fb4c8d5-qqh27\" (UID: \"86e298cc-dd62-489e-9cb2-180bb03d395e\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"
May 20 14:00:42.301955 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:42.301859    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"
May 20 14:00:42.454926 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:42.454891    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"]
May 20 14:00:42.456881 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:42.456848    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e298cc_dd62_489e_9cb2_180bb03d395e.slice/crio-75f7ddb63379da90e20d6fb20f5b92f612af27e28834831bb8797e0caef86350 WatchSource:0}: Error finding container 75f7ddb63379da90e20d6fb20f5b92f612af27e28834831bb8797e0caef86350: Status 404 returned error can't find the container with id 75f7ddb63379da90e20d6fb20f5b92f612af27e28834831bb8797e0caef86350
May 20 14:00:43.365829 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:43.365794    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27" event={"ID":"86e298cc-dd62-489e-9cb2-180bb03d395e","Type":"ContainerStarted","Data":"75f7ddb63379da90e20d6fb20f5b92f612af27e28834831bb8797e0caef86350"}
May 20 14:00:44.187645 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.187609    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"]
May 20 14:00:44.192391 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.192363    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"
May 20 14:00:44.195865 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.195611    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
May 20 14:00:44.195865 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.195791    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-d55zm\""
May 20 14:00:44.204410 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.204374    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"]
May 20 14:00:44.292158 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.292112    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8ng\" (UniqueName: \"kubernetes.io/projected/9901f55d-73bd-4c37-b389-05ba74718cc0-kube-api-access-lm8ng\") pod \"dns-operator-controller-manager-844548ff4c-k8v9v\" (UID: \"9901f55d-73bd-4c37-b389-05ba74718cc0\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"
May 20 14:00:44.357621 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.357572    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7457dcf47b-gxczx" podUID="1ffe2955-cb91-4abc-9b1d-ad5b1c113980" containerName="console" containerID="cri-o://d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d" gracePeriod=15
May 20 14:00:44.393312 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.393271    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8ng\" (UniqueName: \"kubernetes.io/projected/9901f55d-73bd-4c37-b389-05ba74718cc0-kube-api-access-lm8ng\") pod \"dns-operator-controller-manager-844548ff4c-k8v9v\" (UID: \"9901f55d-73bd-4c37-b389-05ba74718cc0\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"
May 20 14:00:44.416539 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.416501    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8ng\" (UniqueName: \"kubernetes.io/projected/9901f55d-73bd-4c37-b389-05ba74718cc0-kube-api-access-lm8ng\") pod \"dns-operator-controller-manager-844548ff4c-k8v9v\" (UID: \"9901f55d-73bd-4c37-b389-05ba74718cc0\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"
May 20 14:00:44.509582 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:44.509490    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"
May 20 14:00:45.178322 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.178292    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7457dcf47b-gxczx_1ffe2955-cb91-4abc-9b1d-ad5b1c113980/console/0.log"
May 20 14:00:45.178450 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.178375    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 14:00:45.300024 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.299995    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"]
May 20 14:00:45.302067 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:00:45.302036    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9901f55d_73bd_4c37_b389_05ba74718cc0.slice/crio-e8daa2052423a7630d692a0052e10d9f4ef96753f91c40d8c8c8cfcfeef6e052 WatchSource:0}: Error finding container e8daa2052423a7630d692a0052e10d9f4ef96753f91c40d8c8c8cfcfeef6e052: Status 404 returned error can't find the container with id e8daa2052423a7630d692a0052e10d9f4ef96753f91c40d8c8c8cfcfeef6e052
May 20 14:00:45.303182 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303160    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-config\") pod \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") "
May 20 14:00:45.303283 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303201    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-service-ca\") pod \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") "
May 20 14:00:45.303283 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303243    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsjz6\" (UniqueName: \"kubernetes.io/projected/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-kube-api-access-tsjz6\") pod \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") "
May 20 14:00:45.303283 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303277    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-trusted-ca-bundle\") pod \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") "
May 20 14:00:45.303449 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303325    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-serving-cert\") pod \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") "
May 20 14:00:45.303449 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303358    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-oauth-serving-cert\") pod \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") "
May 20 14:00:45.303543 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303447    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-oauth-config\") pod \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\" (UID: \"1ffe2955-cb91-4abc-9b1d-ad5b1c113980\") "
May 20 14:00:45.303593 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303545    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-service-ca" (OuterVolumeSpecName: "service-ca") pod "1ffe2955-cb91-4abc-9b1d-ad5b1c113980" (UID: "1ffe2955-cb91-4abc-9b1d-ad5b1c113980"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:00:45.303862 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.303809    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1ffe2955-cb91-4abc-9b1d-ad5b1c113980" (UID: "1ffe2955-cb91-4abc-9b1d-ad5b1c113980"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:00:45.304037 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.304002    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1ffe2955-cb91-4abc-9b1d-ad5b1c113980" (UID: "1ffe2955-cb91-4abc-9b1d-ad5b1c113980"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:00:45.304037 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.304019    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-config" (OuterVolumeSpecName: "console-config") pod "1ffe2955-cb91-4abc-9b1d-ad5b1c113980" (UID: "1ffe2955-cb91-4abc-9b1d-ad5b1c113980"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:00:45.304421 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.304320    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:45.304421 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.304342    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-service-ca\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:45.304421 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.304357    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-trusted-ca-bundle\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:45.304421 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.304372    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-oauth-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:45.305926 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.305894    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-kube-api-access-tsjz6" (OuterVolumeSpecName: "kube-api-access-tsjz6") pod "1ffe2955-cb91-4abc-9b1d-ad5b1c113980" (UID: "1ffe2955-cb91-4abc-9b1d-ad5b1c113980"). InnerVolumeSpecName "kube-api-access-tsjz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:00:45.306027 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.306004    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1ffe2955-cb91-4abc-9b1d-ad5b1c113980" (UID: "1ffe2955-cb91-4abc-9b1d-ad5b1c113980"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:00:45.306067 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.306026    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1ffe2955-cb91-4abc-9b1d-ad5b1c113980" (UID: "1ffe2955-cb91-4abc-9b1d-ad5b1c113980"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:00:45.375637 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.375595    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27" event={"ID":"86e298cc-dd62-489e-9cb2-180bb03d395e","Type":"ContainerStarted","Data":"f0d7f7a5355334a6f8c7fd9aad0eeb3ba58247256bc27de7326305d5abcfb912"}
May 20 14:00:45.375865 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.375661    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"
May 20 14:00:45.376867 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.376829    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7457dcf47b-gxczx_1ffe2955-cb91-4abc-9b1d-ad5b1c113980/console/0.log"
May 20 14:00:45.376955 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.376882    2554 generic.go:358] "Generic (PLEG): container finished" podID="1ffe2955-cb91-4abc-9b1d-ad5b1c113980" containerID="d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d" exitCode=2
May 20 14:00:45.376955 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.376908    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457dcf47b-gxczx" event={"ID":"1ffe2955-cb91-4abc-9b1d-ad5b1c113980","Type":"ContainerDied","Data":"d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d"}
May 20 14:00:45.376955 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.376931    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7457dcf47b-gxczx"
May 20 14:00:45.376955 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.376939    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7457dcf47b-gxczx" event={"ID":"1ffe2955-cb91-4abc-9b1d-ad5b1c113980","Type":"ContainerDied","Data":"ef4bff60cb7be75924edfe389238639645914f4cf60de872ed24bd2baf6dd2aa"}
May 20 14:00:45.376955 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.376956    2554 scope.go:117] "RemoveContainer" containerID="d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d"
May 20 14:00:45.378250 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.378231    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v" event={"ID":"9901f55d-73bd-4c37-b389-05ba74718cc0","Type":"ContainerStarted","Data":"e8daa2052423a7630d692a0052e10d9f4ef96753f91c40d8c8c8cfcfeef6e052"}
May 20 14:00:45.385858 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.385808    2554 scope.go:117] "RemoveContainer" containerID="d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d"
May 20 14:00:45.386113 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:00:45.386095    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d\": container with ID starting with d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d not found: ID does not exist" containerID="d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d"
May 20 14:00:45.386170 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.386122    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d"} err="failed to get container status \"d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d\": rpc error: code = NotFound desc = could not find container \"d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d\": container with ID starting with d1754116811660b35913e2451709d920e5d6c33908a911bff9fd26cdb715ee1d not found: ID does not exist"
May 20 14:00:45.397210 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.397154    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27" podStartSLOduration=1.64909785 podStartE2EDuration="4.397136375s" podCreationTimestamp="2026-05-20 14:00:41 +0000 UTC" firstStartedPulling="2026-05-20 14:00:42.458871995 +0000 UTC m=+627.081597041" lastFinishedPulling="2026-05-20 14:00:45.206910516 +0000 UTC m=+629.829635566" observedRunningTime="2026-05-20 14:00:45.396335179 +0000 UTC m=+630.019060248" watchObservedRunningTime="2026-05-20 14:00:45.397136375 +0000 UTC m=+630.019861444"
May 20 14:00:45.405117 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.405090    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-oauth-config\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:45.405117 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.405118    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjz6\" (UniqueName: \"kubernetes.io/projected/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-kube-api-access-tsjz6\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:45.405252 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.405128    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ffe2955-cb91-4abc-9b1d-ad5b1c113980-console-serving-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:00:45.414106 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.414043    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7457dcf47b-gxczx"]
May 20 14:00:45.418774 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:45.418749    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7457dcf47b-gxczx"]
May 20 14:00:46.178713 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:46.178678    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ffe2955-cb91-4abc-9b1d-ad5b1c113980" path="/var/lib/kubelet/pods/1ffe2955-cb91-4abc-9b1d-ad5b1c113980/volumes"
May 20 14:00:48.392829 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:48.392732    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v" event={"ID":"9901f55d-73bd-4c37-b389-05ba74718cc0","Type":"ContainerStarted","Data":"e696e681eeef40036b9f19d9ac3dd15b103be60b0179c8f58fd71e81a5c8026a"}
May 20 14:00:48.393295 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:48.392866    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"
May 20 14:00:48.415454 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:48.415404    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v" podStartSLOduration=1.763216253 podStartE2EDuration="4.415386845s" podCreationTimestamp="2026-05-20 14:00:44 +0000 UTC" firstStartedPulling="2026-05-20 14:00:45.304895809 +0000 UTC m=+629.927620857" lastFinishedPulling="2026-05-20 14:00:47.9570664 +0000 UTC m=+632.579791449" observedRunningTime="2026-05-20 14:00:48.412412598 +0000 UTC m=+633.035137670" watchObservedRunningTime="2026-05-20 14:00:48.415386845 +0000 UTC m=+633.038111913"
May 20 14:00:52.363310 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:52.363279    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-z2bt4"
May 20 14:00:56.385176 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:56.385146    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-qqh27"
May 20 14:00:59.399516 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:00:59.399483    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-k8v9v"
May 20 14:01:26.156781 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.156744    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2nm2m"]
May 20 14:01:26.157246 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.157114    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ffe2955-cb91-4abc-9b1d-ad5b1c113980" containerName="console"
May 20 14:01:26.157246 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.157126    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ffe2955-cb91-4abc-9b1d-ad5b1c113980" containerName="console"
May 20 14:01:26.157246 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.157199    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ffe2955-cb91-4abc-9b1d-ad5b1c113980" containerName="console"
May 20 14:01:26.163631 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.163600    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.166214 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.166188    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-25lfs\""
May 20 14:01:26.166355 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.166189    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 20 14:01:26.167160 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.167131    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2nm2m"]
May 20 14:01:26.198045 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.198010    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2nm2m"]
May 20 14:01:26.272221 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.272182    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/42ff960b-532e-4f74-ae08-5277b8808aac-config-file\") pod \"limitador-limitador-67566c68b4-2nm2m\" (UID: \"42ff960b-532e-4f74-ae08-5277b8808aac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.272399 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.272262    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2qt\" (UniqueName: \"kubernetes.io/projected/42ff960b-532e-4f74-ae08-5277b8808aac-kube-api-access-qt2qt\") pod \"limitador-limitador-67566c68b4-2nm2m\" (UID: \"42ff960b-532e-4f74-ae08-5277b8808aac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.373675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.373635    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/42ff960b-532e-4f74-ae08-5277b8808aac-config-file\") pod \"limitador-limitador-67566c68b4-2nm2m\" (UID: \"42ff960b-532e-4f74-ae08-5277b8808aac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.373905 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.373697    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2qt\" (UniqueName: \"kubernetes.io/projected/42ff960b-532e-4f74-ae08-5277b8808aac-kube-api-access-qt2qt\") pod \"limitador-limitador-67566c68b4-2nm2m\" (UID: \"42ff960b-532e-4f74-ae08-5277b8808aac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.374359 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.374338    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/42ff960b-532e-4f74-ae08-5277b8808aac-config-file\") pod \"limitador-limitador-67566c68b4-2nm2m\" (UID: \"42ff960b-532e-4f74-ae08-5277b8808aac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.382515 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.382485    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2qt\" (UniqueName: \"kubernetes.io/projected/42ff960b-532e-4f74-ae08-5277b8808aac-kube-api-access-qt2qt\") pod \"limitador-limitador-67566c68b4-2nm2m\" (UID: \"42ff960b-532e-4f74-ae08-5277b8808aac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.475090 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.474993    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:26.620101 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.620071    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2nm2m"]
May 20 14:01:26.621145 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:01:26.621119    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ff960b_532e_4f74_ae08_5277b8808aac.slice/crio-e490528bf6efaca4f0ef44e91d346f06d426dfb57a0abc0e9c0a1f0eaf2d181f WatchSource:0}: Error finding container e490528bf6efaca4f0ef44e91d346f06d426dfb57a0abc0e9c0a1f0eaf2d181f: Status 404 returned error can't find the container with id e490528bf6efaca4f0ef44e91d346f06d426dfb57a0abc0e9c0a1f0eaf2d181f
May 20 14:01:26.623090 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:26.623072    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:01:27.551413 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:27.551373    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m" event={"ID":"42ff960b-532e-4f74-ae08-5277b8808aac","Type":"ContainerStarted","Data":"e490528bf6efaca4f0ef44e91d346f06d426dfb57a0abc0e9c0a1f0eaf2d181f"}
May 20 14:01:31.572753 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:31.572712    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m" event={"ID":"42ff960b-532e-4f74-ae08-5277b8808aac","Type":"ContainerStarted","Data":"aa0dcd506c544072af2b20f1a8cae2b42161e49c186a85ed29d423330c3f34bb"}
May 20 14:01:31.573237 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:31.572835    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:01:31.591487 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:31.591434    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m" podStartSLOduration=1.606392324 podStartE2EDuration="5.59141837s" podCreationTimestamp="2026-05-20 14:01:26 +0000 UTC" firstStartedPulling="2026-05-20 14:01:26.623199201 +0000 UTC m=+671.245924246" lastFinishedPulling="2026-05-20 14:01:30.608225231 +0000 UTC m=+675.230950292" observedRunningTime="2026-05-20 14:01:31.589746825 +0000 UTC m=+676.212471893" watchObservedRunningTime="2026-05-20 14:01:31.59141837 +0000 UTC m=+676.214143437"
May 20 14:01:42.579299 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:01:42.579265    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-2nm2m"
May 20 14:02:06.729088 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:06.728994    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"]
May 20 14:02:06.729688 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:06.729411    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x" podUID="b23ad75c-59c6-4a8b-a472-1bc320e04cf9" containerName="discovery" containerID="cri-o://3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b" gracePeriod=30
May 20 14:02:06.998099 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:06.998063    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:02:07.036801 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.036771    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-ca-configmap\") pod \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") "
May 20 14:02:07.036801 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.036808    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-kubeconfig\") pod \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") "
May 20 14:02:07.037043 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.036835    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxb5v\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-kube-api-access-lxb5v\") pod \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") "
May 20 14:02:07.037043 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.036900    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-local-certs\") pod \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") "
May 20 14:02:07.037043 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.036948    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-dns-cert\") pod \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") "
May 20 14:02:07.037043 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.036998    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-token\") pod \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") "
May 20 14:02:07.037258 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.037049    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-cacerts\") pod \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\" (UID: \"b23ad75c-59c6-4a8b-a472-1bc320e04cf9\") "
May 20 14:02:07.037258 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.037236    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b23ad75c-59c6-4a8b-a472-1bc320e04cf9" (UID: "b23ad75c-59c6-4a8b-a472-1bc320e04cf9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:02:07.037376 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.037346    2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-ca-configmap\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:07.039791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.039708    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-local-certs" (OuterVolumeSpecName: "local-certs") pod "b23ad75c-59c6-4a8b-a472-1bc320e04cf9" (UID: "b23ad75c-59c6-4a8b-a472-1bc320e04cf9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:02:07.039791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.039724    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-cacerts" (OuterVolumeSpecName: "cacerts") pod "b23ad75c-59c6-4a8b-a472-1bc320e04cf9" (UID: "b23ad75c-59c6-4a8b-a472-1bc320e04cf9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:02:07.039791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.039751    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-kube-api-access-lxb5v" (OuterVolumeSpecName: "kube-api-access-lxb5v") pod "b23ad75c-59c6-4a8b-a472-1bc320e04cf9" (UID: "b23ad75c-59c6-4a8b-a472-1bc320e04cf9"). InnerVolumeSpecName "kube-api-access-lxb5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:02:07.040044 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.039886    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-token" (OuterVolumeSpecName: "istio-token") pod "b23ad75c-59c6-4a8b-a472-1bc320e04cf9" (UID: "b23ad75c-59c6-4a8b-a472-1bc320e04cf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:02:07.040368 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.040347    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b23ad75c-59c6-4a8b-a472-1bc320e04cf9" (UID: "b23ad75c-59c6-4a8b-a472-1bc320e04cf9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:02:07.040832 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.040816    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b23ad75c-59c6-4a8b-a472-1bc320e04cf9" (UID: "b23ad75c-59c6-4a8b-a472-1bc320e04cf9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:02:07.138531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.138481    2554 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-cacerts\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:07.138531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.138515    2554 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-kubeconfig\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:07.138531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.138526    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxb5v\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-kube-api-access-lxb5v\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:07.138531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.138536    2554 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-local-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:07.138531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.138546    2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-csr-dns-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:07.138958 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.138554    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b23ad75c-59c6-4a8b-a472-1bc320e04cf9-istio-token\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:07.728027 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.727989    2554 generic.go:358] "Generic (PLEG): container finished" podID="b23ad75c-59c6-4a8b-a472-1bc320e04cf9" containerID="3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b" exitCode=0
May 20 14:02:07.728217 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.728045    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"
May 20 14:02:07.728217 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.728075    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x" event={"ID":"b23ad75c-59c6-4a8b-a472-1bc320e04cf9","Type":"ContainerDied","Data":"3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b"}
May 20 14:02:07.728217 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.728121    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x" event={"ID":"b23ad75c-59c6-4a8b-a472-1bc320e04cf9","Type":"ContainerDied","Data":"3e3866638038d8af07aaef88c44a139c298d53fdc12998a153c3e893fd8335b0"}
May 20 14:02:07.728217 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.728141    2554 scope.go:117] "RemoveContainer" containerID="3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b"
May 20 14:02:07.739091 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.739058    2554 scope.go:117] "RemoveContainer" containerID="3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b"
May 20 14:02:07.744857 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:02:07.739631    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b\": container with ID starting with 3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b not found: ID does not exist" containerID="3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b"
May 20 14:02:07.744857 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.739677    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b"} err="failed to get container status \"3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b\": rpc error: code = NotFound desc = could not find container \"3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b\": container with ID starting with 3ad46cb72346b1eeb8455d13b92f2a01775eba1032c81f020e14a9dba483c07b not found: ID does not exist"
May 20 14:02:07.750971 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.750942    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"]
May 20 14:02:07.757629 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:07.757601    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-767cf5cf55-zzv8x"]
May 20 14:02:08.176856 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:08.176813    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b23ad75c-59c6-4a8b-a472-1bc320e04cf9" path="/var/lib/kubelet/pods/b23ad75c-59c6-4a8b-a472-1bc320e04cf9/volumes"
May 20 14:02:10.476996 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.476954    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867cffc775-p2fdx"]
May 20 14:02:10.477501 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.477487    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23ad75c-59c6-4a8b-a472-1bc320e04cf9" containerName="discovery"
May 20 14:02:10.477554 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.477504    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23ad75c-59c6-4a8b-a472-1bc320e04cf9" containerName="discovery"
May 20 14:02:10.477632 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.477620    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="b23ad75c-59c6-4a8b-a472-1bc320e04cf9" containerName="discovery"
May 20 14:02:10.482262 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.482237    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.484971 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.484945    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-g6g7n\""
May 20 14:02:10.484971 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.484957    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 14:02:10.485287 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.485083    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 14:02:10.485287 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.485192    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 20 14:02:10.493912 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.493880    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-p2fdx"]
May 20 14:02:10.497682 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.497652    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"]
May 20 14:02:10.501905 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.501883    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.505751 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.505727    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 20 14:02:10.506626 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.506604    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-jqwfs\""
May 20 14:02:10.515402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.515370    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"]
May 20 14:02:10.522063 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.522040    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-5tfqj"]
May 20 14:02:10.525579 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.525559    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.528347 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.528330    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 14:02:10.528855 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.528820    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9pmgg\""
May 20 14:02:10.536683 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.536652    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-5tfqj"]
May 20 14:02:10.574856 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.574802    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gnz\" (UniqueName: \"kubernetes.io/projected/a9b76b23-f50a-4ee6-b68a-478792858200-kube-api-access-84gnz\") pod \"kserve-controller-manager-867cffc775-p2fdx\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") " pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.575066 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.574902    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b76b23-f50a-4ee6-b68a-478792858200-cert\") pod \"kserve-controller-manager-867cffc775-p2fdx\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") " pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.575066 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.574968    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d754395-29c2-40a1-864b-577565d9acb4-data\") pod \"seaweedfs-79f457656b-5tfqj\" (UID: \"2d754395-29c2-40a1-864b-577565d9acb4\") " pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.575189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.575039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-cert\") pod \"llmisvc-controller-manager-58b7b7548c-wgtk6\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") " pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.575189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.575090    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2jj\" (UniqueName: \"kubernetes.io/projected/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-kube-api-access-nb2jj\") pod \"llmisvc-controller-manager-58b7b7548c-wgtk6\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") " pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.575189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.575144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpq5m\" (UniqueName: \"kubernetes.io/projected/2d754395-29c2-40a1-864b-577565d9acb4-kube-api-access-zpq5m\") pod \"seaweedfs-79f457656b-5tfqj\" (UID: \"2d754395-29c2-40a1-864b-577565d9acb4\") " pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.676455 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.676422    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpq5m\" (UniqueName: \"kubernetes.io/projected/2d754395-29c2-40a1-864b-577565d9acb4-kube-api-access-zpq5m\") pod \"seaweedfs-79f457656b-5tfqj\" (UID: \"2d754395-29c2-40a1-864b-577565d9acb4\") " pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.676651 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.676472    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-84gnz\" (UniqueName: \"kubernetes.io/projected/a9b76b23-f50a-4ee6-b68a-478792858200-kube-api-access-84gnz\") pod \"kserve-controller-manager-867cffc775-p2fdx\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") " pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.676651 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.676512    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b76b23-f50a-4ee6-b68a-478792858200-cert\") pod \"kserve-controller-manager-867cffc775-p2fdx\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") " pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.676651 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.676555    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d754395-29c2-40a1-864b-577565d9acb4-data\") pod \"seaweedfs-79f457656b-5tfqj\" (UID: \"2d754395-29c2-40a1-864b-577565d9acb4\") " pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.676651 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.676613    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-cert\") pod \"llmisvc-controller-manager-58b7b7548c-wgtk6\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") " pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.676651 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.676638    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2jj\" (UniqueName: \"kubernetes.io/projected/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-kube-api-access-nb2jj\") pod \"llmisvc-controller-manager-58b7b7548c-wgtk6\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") " pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.677078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.677054    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2d754395-29c2-40a1-864b-577565d9acb4-data\") pod \"seaweedfs-79f457656b-5tfqj\" (UID: \"2d754395-29c2-40a1-864b-577565d9acb4\") " pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.679248 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.679226    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-cert\") pod \"llmisvc-controller-manager-58b7b7548c-wgtk6\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") " pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.679345 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.679317    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b76b23-f50a-4ee6-b68a-478792858200-cert\") pod \"kserve-controller-manager-867cffc775-p2fdx\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") " pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.684464 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.684430    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gnz\" (UniqueName: \"kubernetes.io/projected/a9b76b23-f50a-4ee6-b68a-478792858200-kube-api-access-84gnz\") pod \"kserve-controller-manager-867cffc775-p2fdx\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") " pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.684918 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.684893    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpq5m\" (UniqueName: \"kubernetes.io/projected/2d754395-29c2-40a1-864b-577565d9acb4-kube-api-access-zpq5m\") pod \"seaweedfs-79f457656b-5tfqj\" (UID: \"2d754395-29c2-40a1-864b-577565d9acb4\") " pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.684972 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.684902    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2jj\" (UniqueName: \"kubernetes.io/projected/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-kube-api-access-nb2jj\") pod \"llmisvc-controller-manager-58b7b7548c-wgtk6\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") " pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.796832 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.796743    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:10.814310 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.814278    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:10.838390 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.838357    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:10.970676 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.970631    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-p2fdx"]
May 20 14:02:10.971848 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:02:10.971809    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b76b23_f50a_4ee6_b68a_478792858200.slice/crio-4962aa9a42d1b1cdbcb223d1e5a5514980b93394326ed86862035295f6f57969 WatchSource:0}: Error finding container 4962aa9a42d1b1cdbcb223d1e5a5514980b93394326ed86862035295f6f57969: Status 404 returned error can't find the container with id 4962aa9a42d1b1cdbcb223d1e5a5514980b93394326ed86862035295f6f57969
May 20 14:02:10.997061 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:10.997029    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"]
May 20 14:02:10.998434 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:02:10.998392    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e03b710_ae09_4591_bc03_ce91f5ac3ec1.slice/crio-67aa2035b9d1ac50463c52076640db3738c65d173239b4e81d32b655a93844e4 WatchSource:0}: Error finding container 67aa2035b9d1ac50463c52076640db3738c65d173239b4e81d32b655a93844e4: Status 404 returned error can't find the container with id 67aa2035b9d1ac50463c52076640db3738c65d173239b4e81d32b655a93844e4
May 20 14:02:11.022557 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:11.022528    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-5tfqj"]
May 20 14:02:11.023807 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:02:11.023781    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d754395_29c2_40a1_864b_577565d9acb4.slice/crio-48272cb783c11e78b439b9ba35fd74af992ff77871c7b2fb8a2b104d72b7cafb WatchSource:0}: Error finding container 48272cb783c11e78b439b9ba35fd74af992ff77871c7b2fb8a2b104d72b7cafb: Status 404 returned error can't find the container with id 48272cb783c11e78b439b9ba35fd74af992ff77871c7b2fb8a2b104d72b7cafb
May 20 14:02:11.759557 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:11.759506    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-p2fdx" event={"ID":"a9b76b23-f50a-4ee6-b68a-478792858200","Type":"ContainerStarted","Data":"4962aa9a42d1b1cdbcb223d1e5a5514980b93394326ed86862035295f6f57969"}
May 20 14:02:11.762316 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:11.762281    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6" event={"ID":"4e03b710-ae09-4591-bc03-ce91f5ac3ec1","Type":"ContainerStarted","Data":"67aa2035b9d1ac50463c52076640db3738c65d173239b4e81d32b655a93844e4"}
May 20 14:02:11.764154 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:11.764110    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-5tfqj" event={"ID":"2d754395-29c2-40a1-864b-577565d9acb4","Type":"ContainerStarted","Data":"48272cb783c11e78b439b9ba35fd74af992ff77871c7b2fb8a2b104d72b7cafb"}
May 20 14:02:16.791256 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.791217    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-p2fdx" event={"ID":"a9b76b23-f50a-4ee6-b68a-478792858200","Type":"ContainerStarted","Data":"663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553"}
May 20 14:02:16.791763 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.791334    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:16.792606 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.792577    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6" event={"ID":"4e03b710-ae09-4591-bc03-ce91f5ac3ec1","Type":"ContainerStarted","Data":"2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb"}
May 20 14:02:16.792724 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.792678    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:16.793883 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.793829    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-5tfqj" event={"ID":"2d754395-29c2-40a1-864b-577565d9acb4","Type":"ContainerStarted","Data":"3f90477b2c457a97857d26a9c333a14d05bd2136af9d902b07dba5e240dc2ce4"}
May 20 14:02:16.793994 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.793903    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:16.808779 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.808721    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867cffc775-p2fdx" podStartSLOduration=2.827004036 podStartE2EDuration="6.808705799s" podCreationTimestamp="2026-05-20 14:02:10 +0000 UTC" firstStartedPulling="2026-05-20 14:02:10.973687967 +0000 UTC m=+715.596413012" lastFinishedPulling="2026-05-20 14:02:14.955389729 +0000 UTC m=+719.578114775" observedRunningTime="2026-05-20 14:02:16.807775399 +0000 UTC m=+721.430500478" watchObservedRunningTime="2026-05-20 14:02:16.808705799 +0000 UTC m=+721.431430869"
May 20 14:02:16.830423 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.830366    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6" podStartSLOduration=2.067754696 podStartE2EDuration="6.830350966s" podCreationTimestamp="2026-05-20 14:02:10 +0000 UTC" firstStartedPulling="2026-05-20 14:02:11.000207359 +0000 UTC m=+715.622932420" lastFinishedPulling="2026-05-20 14:02:15.762803645 +0000 UTC m=+720.385528690" observedRunningTime="2026-05-20 14:02:16.829604336 +0000 UTC m=+721.452329407" watchObservedRunningTime="2026-05-20 14:02:16.830350966 +0000 UTC m=+721.453076033"
May 20 14:02:16.846934 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:16.846877    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-5tfqj" podStartSLOduration=2.065657877 podStartE2EDuration="6.846860849s" podCreationTimestamp="2026-05-20 14:02:10 +0000 UTC" firstStartedPulling="2026-05-20 14:02:11.025084242 +0000 UTC m=+715.647809287" lastFinishedPulling="2026-05-20 14:02:15.806287211 +0000 UTC m=+720.429012259" observedRunningTime="2026-05-20 14:02:16.8440431 +0000 UTC m=+721.466768167" watchObservedRunningTime="2026-05-20 14:02:16.846860849 +0000 UTC m=+721.469585917"
May 20 14:02:22.800594 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:22.800555    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-5tfqj"
May 20 14:02:47.801584 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:47.801542    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:02:47.804583 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:47.804558    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:49.350061 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.350017    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-p2fdx"]
May 20 14:02:49.350543 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.350295    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-867cffc775-p2fdx" podUID="a9b76b23-f50a-4ee6-b68a-478792858200" containerName="manager" containerID="cri-o://663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553" gracePeriod=10
May 20 14:02:49.375828 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.375796    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867cffc775-ndt9f"]
May 20 14:02:49.378280 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.378254    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.389958 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.389927    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-ndt9f"]
May 20 14:02:49.426014 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.425979    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810eb18b-874f-42d5-bb41-ebc13e8df7c2-cert\") pod \"kserve-controller-manager-867cffc775-ndt9f\" (UID: \"810eb18b-874f-42d5-bb41-ebc13e8df7c2\") " pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.426200 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.426048    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmpn\" (UniqueName: \"kubernetes.io/projected/810eb18b-874f-42d5-bb41-ebc13e8df7c2-kube-api-access-gvmpn\") pod \"kserve-controller-manager-867cffc775-ndt9f\" (UID: \"810eb18b-874f-42d5-bb41-ebc13e8df7c2\") " pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.526586 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.526547    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810eb18b-874f-42d5-bb41-ebc13e8df7c2-cert\") pod \"kserve-controller-manager-867cffc775-ndt9f\" (UID: \"810eb18b-874f-42d5-bb41-ebc13e8df7c2\") " pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.526766 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.526639    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmpn\" (UniqueName: \"kubernetes.io/projected/810eb18b-874f-42d5-bb41-ebc13e8df7c2-kube-api-access-gvmpn\") pod \"kserve-controller-manager-867cffc775-ndt9f\" (UID: \"810eb18b-874f-42d5-bb41-ebc13e8df7c2\") " pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.529549 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.529520    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810eb18b-874f-42d5-bb41-ebc13e8df7c2-cert\") pod \"kserve-controller-manager-867cffc775-ndt9f\" (UID: \"810eb18b-874f-42d5-bb41-ebc13e8df7c2\") " pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.535113 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.535089    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmpn\" (UniqueName: \"kubernetes.io/projected/810eb18b-874f-42d5-bb41-ebc13e8df7c2-kube-api-access-gvmpn\") pod \"kserve-controller-manager-867cffc775-ndt9f\" (UID: \"810eb18b-874f-42d5-bb41-ebc13e8df7c2\") " pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.602944 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.602864    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:49.627576 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.627549    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84gnz\" (UniqueName: \"kubernetes.io/projected/a9b76b23-f50a-4ee6-b68a-478792858200-kube-api-access-84gnz\") pod \"a9b76b23-f50a-4ee6-b68a-478792858200\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") "
May 20 14:02:49.627759 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.627635    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b76b23-f50a-4ee6-b68a-478792858200-cert\") pod \"a9b76b23-f50a-4ee6-b68a-478792858200\" (UID: \"a9b76b23-f50a-4ee6-b68a-478792858200\") "
May 20 14:02:49.630014 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.629980    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b76b23-f50a-4ee6-b68a-478792858200-cert" (OuterVolumeSpecName: "cert") pod "a9b76b23-f50a-4ee6-b68a-478792858200" (UID: "a9b76b23-f50a-4ee6-b68a-478792858200"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:02:49.630501 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.630473    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b76b23-f50a-4ee6-b68a-478792858200-kube-api-access-84gnz" (OuterVolumeSpecName: "kube-api-access-84gnz") pod "a9b76b23-f50a-4ee6-b68a-478792858200" (UID: "a9b76b23-f50a-4ee6-b68a-478792858200"). InnerVolumeSpecName "kube-api-access-84gnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:02:49.728868 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.728808    2554 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b76b23-f50a-4ee6-b68a-478792858200-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:49.728868 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.728862    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84gnz\" (UniqueName: \"kubernetes.io/projected/a9b76b23-f50a-4ee6-b68a-478792858200-kube-api-access-84gnz\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:02:49.747819 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.747765    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:49.880402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.880377    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-ndt9f"]
May 20 14:02:49.882217 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:02:49.882183    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810eb18b_874f_42d5_bb41_ebc13e8df7c2.slice/crio-414983eb86df79ae4bcd0b8556ea4bf6763c1930a548b7e84983c257f84a704a WatchSource:0}: Error finding container 414983eb86df79ae4bcd0b8556ea4bf6763c1930a548b7e84983c257f84a704a: Status 404 returned error can't find the container with id 414983eb86df79ae4bcd0b8556ea4bf6763c1930a548b7e84983c257f84a704a
May 20 14:02:49.940534 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.940498    2554 generic.go:358] "Generic (PLEG): container finished" podID="a9b76b23-f50a-4ee6-b68a-478792858200" containerID="663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553" exitCode=0
May 20 14:02:49.940721 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.940558    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867cffc775-p2fdx"
May 20 14:02:49.940721 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.940569    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-p2fdx" event={"ID":"a9b76b23-f50a-4ee6-b68a-478792858200","Type":"ContainerDied","Data":"663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553"}
May 20 14:02:49.940721 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.940607    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-p2fdx" event={"ID":"a9b76b23-f50a-4ee6-b68a-478792858200","Type":"ContainerDied","Data":"4962aa9a42d1b1cdbcb223d1e5a5514980b93394326ed86862035295f6f57969"}
May 20 14:02:49.940721 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.940624    2554 scope.go:117] "RemoveContainer" containerID="663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553"
May 20 14:02:49.941671 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.941652    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-ndt9f" event={"ID":"810eb18b-874f-42d5-bb41-ebc13e8df7c2","Type":"ContainerStarted","Data":"414983eb86df79ae4bcd0b8556ea4bf6763c1930a548b7e84983c257f84a704a"}
May 20 14:02:49.949658 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.949641    2554 scope.go:117] "RemoveContainer" containerID="663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553"
May 20 14:02:49.949934 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:02:49.949914    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553\": container with ID starting with 663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553 not found: ID does not exist" containerID="663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553"
May 20 14:02:49.950017 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.949941    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553"} err="failed to get container status \"663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553\": rpc error: code = NotFound desc = could not find container \"663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553\": container with ID starting with 663df66cab0316340735b334d0152d878884ba6486abb9b55659101d7be43553 not found: ID does not exist"
May 20 14:02:49.962741 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.962710    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-p2fdx"]
May 20 14:02:49.966235 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:49.966207    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-867cffc775-p2fdx"]
May 20 14:02:50.177441 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:50.177405    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b76b23-f50a-4ee6-b68a-478792858200" path="/var/lib/kubelet/pods/a9b76b23-f50a-4ee6-b68a-478792858200/volumes"
May 20 14:02:50.948281 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:50.948242    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867cffc775-ndt9f" event={"ID":"810eb18b-874f-42d5-bb41-ebc13e8df7c2","Type":"ContainerStarted","Data":"e2e89c3505c81d471127318f1592a3649c839d5192037f6a9705b8e0b12a6661"}
May 20 14:02:50.948753 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:50.948361    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:02:50.965937 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:02:50.965886    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867cffc775-ndt9f" podStartSLOduration=1.5545332090000001 podStartE2EDuration="1.965871499s" podCreationTimestamp="2026-05-20 14:02:49 +0000 UTC" firstStartedPulling="2026-05-20 14:02:49.883492986 +0000 UTC m=+754.506218032" lastFinishedPulling="2026-05-20 14:02:50.294831273 +0000 UTC m=+754.917556322" observedRunningTime="2026-05-20 14:02:50.96399257 +0000 UTC m=+755.586717631" watchObservedRunningTime="2026-05-20 14:02:50.965871499 +0000 UTC m=+755.588596567"
May 20 14:03:21.957588 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:21.957556    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867cffc775-ndt9f"
May 20 14:03:22.921715 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.921666    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-zvn8b"]
May 20 14:03:22.922293 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.922256    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b76b23-f50a-4ee6-b68a-478792858200" containerName="manager"
May 20 14:03:22.922293 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.922281    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b76b23-f50a-4ee6-b68a-478792858200" containerName="manager"
May 20 14:03:22.922479 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.922394    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9b76b23-f50a-4ee6-b68a-478792858200" containerName="manager"
May 20 14:03:22.925642 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.925624    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:22.930706 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.930680    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kq5dr\""
May 20 14:03:22.930706 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.930681    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 20 14:03:22.940904 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.940870    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-zvn8b"]
May 20 14:03:22.943401 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.943372    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-jzmm8"]
May 20 14:03:22.947226 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.947199    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:22.949863 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.949813    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 20 14:03:22.949963 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.949908    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-q8rrr\""
May 20 14:03:22.955533 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:22.955509    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-jzmm8"]
May 20 14:03:23.030654 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.030617    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-cert\") pod \"odh-model-controller-696fc77849-jzmm8\" (UID: \"b7989cc1-fe4b-458b-9870-3b2ea44fcc94\") " pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.031111 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.030678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qdl\" (UniqueName: \"kubernetes.io/projected/e3570305-84ea-4bb8-a857-370dded78754-kube-api-access-k7qdl\") pod \"model-serving-api-86f7b4b499-zvn8b\" (UID: \"e3570305-84ea-4bb8-a857-370dded78754\") " pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:23.031111 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.030761    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjvl\" (UniqueName: \"kubernetes.io/projected/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-kube-api-access-vhjvl\") pod \"odh-model-controller-696fc77849-jzmm8\" (UID: \"b7989cc1-fe4b-458b-9870-3b2ea44fcc94\") " pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.031111 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.030822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3570305-84ea-4bb8-a857-370dded78754-tls-certs\") pod \"model-serving-api-86f7b4b499-zvn8b\" (UID: \"e3570305-84ea-4bb8-a857-370dded78754\") " pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:23.132293 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.132247    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3570305-84ea-4bb8-a857-370dded78754-tls-certs\") pod \"model-serving-api-86f7b4b499-zvn8b\" (UID: \"e3570305-84ea-4bb8-a857-370dded78754\") " pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:23.132485 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.132366    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-cert\") pod \"odh-model-controller-696fc77849-jzmm8\" (UID: \"b7989cc1-fe4b-458b-9870-3b2ea44fcc94\") " pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.132485 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.132407    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qdl\" (UniqueName: \"kubernetes.io/projected/e3570305-84ea-4bb8-a857-370dded78754-kube-api-access-k7qdl\") pod \"model-serving-api-86f7b4b499-zvn8b\" (UID: \"e3570305-84ea-4bb8-a857-370dded78754\") " pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:23.132561 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:03:23.132508    2554 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 20 14:03:23.132596 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.132549    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjvl\" (UniqueName: \"kubernetes.io/projected/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-kube-api-access-vhjvl\") pod \"odh-model-controller-696fc77849-jzmm8\" (UID: \"b7989cc1-fe4b-458b-9870-3b2ea44fcc94\") " pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.132596 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:03:23.132572    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-cert podName:b7989cc1-fe4b-458b-9870-3b2ea44fcc94 nodeName:}" failed. No retries permitted until 2026-05-20 14:03:23.632555313 +0000 UTC m=+788.255280360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-cert") pod "odh-model-controller-696fc77849-jzmm8" (UID: "b7989cc1-fe4b-458b-9870-3b2ea44fcc94") : secret "odh-model-controller-webhook-cert" not found
May 20 14:03:23.134825 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.134807    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3570305-84ea-4bb8-a857-370dded78754-tls-certs\") pod \"model-serving-api-86f7b4b499-zvn8b\" (UID: \"e3570305-84ea-4bb8-a857-370dded78754\") " pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:23.143203 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.143176    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qdl\" (UniqueName: \"kubernetes.io/projected/e3570305-84ea-4bb8-a857-370dded78754-kube-api-access-k7qdl\") pod \"model-serving-api-86f7b4b499-zvn8b\" (UID: \"e3570305-84ea-4bb8-a857-370dded78754\") " pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:23.145635 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.145608    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjvl\" (UniqueName: \"kubernetes.io/projected/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-kube-api-access-vhjvl\") pod \"odh-model-controller-696fc77849-jzmm8\" (UID: \"b7989cc1-fe4b-458b-9870-3b2ea44fcc94\") " pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.236784 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.236678    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:23.375991 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.375964    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-zvn8b"]
May 20 14:03:23.377961 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:03:23.377931    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3570305_84ea_4bb8_a857_370dded78754.slice/crio-9ffbeaa730f556132f6ec18c81ad2f92625799284dd66f6a3ef2c5e78598bca5 WatchSource:0}: Error finding container 9ffbeaa730f556132f6ec18c81ad2f92625799284dd66f6a3ef2c5e78598bca5: Status 404 returned error can't find the container with id 9ffbeaa730f556132f6ec18c81ad2f92625799284dd66f6a3ef2c5e78598bca5
May 20 14:03:23.636746 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.636650    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-cert\") pod \"odh-model-controller-696fc77849-jzmm8\" (UID: \"b7989cc1-fe4b-458b-9870-3b2ea44fcc94\") " pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.639183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.639155    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7989cc1-fe4b-458b-9870-3b2ea44fcc94-cert\") pod \"odh-model-controller-696fc77849-jzmm8\" (UID: \"b7989cc1-fe4b-458b-9870-3b2ea44fcc94\") " pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.859982 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.859946    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:23.998338 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:23.998292    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-jzmm8"]
May 20 14:03:23.999754 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:03:23.999724    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7989cc1_fe4b_458b_9870_3b2ea44fcc94.slice/crio-62cfcad78ea6d5de040a80058e435f78d2147c012fab578861ad4cd681b094e2 WatchSource:0}: Error finding container 62cfcad78ea6d5de040a80058e435f78d2147c012fab578861ad4cd681b094e2: Status 404 returned error can't find the container with id 62cfcad78ea6d5de040a80058e435f78d2147c012fab578861ad4cd681b094e2
May 20 14:03:24.090444 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:24.090399    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-jzmm8" event={"ID":"b7989cc1-fe4b-458b-9870-3b2ea44fcc94","Type":"ContainerStarted","Data":"62cfcad78ea6d5de040a80058e435f78d2147c012fab578861ad4cd681b094e2"}
May 20 14:03:24.092251 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:24.092217    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-zvn8b" event={"ID":"e3570305-84ea-4bb8-a857-370dded78754","Type":"ContainerStarted","Data":"9ffbeaa730f556132f6ec18c81ad2f92625799284dd66f6a3ef2c5e78598bca5"}
May 20 14:03:28.115268 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:28.115224    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-zvn8b" event={"ID":"e3570305-84ea-4bb8-a857-370dded78754","Type":"ContainerStarted","Data":"46e792502c7cc433fc14cb4ce33026342adcec895b2caecd0a97e574f2d2beb9"}
May 20 14:03:28.115732 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:28.115301    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:28.116653 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:28.116629    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-jzmm8" event={"ID":"b7989cc1-fe4b-458b-9870-3b2ea44fcc94","Type":"ContainerStarted","Data":"0238ba6095c0f1f47c8d4b9809d65278cecc11d5c40463c181355d17f6b1971e"}
May 20 14:03:28.116759 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:28.116735    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:28.134052 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:28.133989    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-zvn8b" podStartSLOduration=2.387171685 podStartE2EDuration="6.133973931s" podCreationTimestamp="2026-05-20 14:03:22 +0000 UTC" firstStartedPulling="2026-05-20 14:03:23.379679439 +0000 UTC m=+788.002404489" lastFinishedPulling="2026-05-20 14:03:27.126481677 +0000 UTC m=+791.749206735" observedRunningTime="2026-05-20 14:03:28.132023311 +0000 UTC m=+792.754748379" watchObservedRunningTime="2026-05-20 14:03:28.133973931 +0000 UTC m=+792.756699006"
May 20 14:03:28.149119 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:28.149061    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-jzmm8" podStartSLOduration=3.01907958 podStartE2EDuration="6.14904049s" podCreationTimestamp="2026-05-20 14:03:22 +0000 UTC" firstStartedPulling="2026-05-20 14:03:24.001492933 +0000 UTC m=+788.624217994" lastFinishedPulling="2026-05-20 14:03:27.131453853 +0000 UTC m=+791.754178904" observedRunningTime="2026-05-20 14:03:28.14791186 +0000 UTC m=+792.770636928" watchObservedRunningTime="2026-05-20 14:03:28.14904049 +0000 UTC m=+792.771765559"
May 20 14:03:39.124422 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:39.124340    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-jzmm8"
May 20 14:03:39.127330 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:39.127308    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-zvn8b"
May 20 14:03:39.954262 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:39.954217    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-5vxwn"]
May 20 14:03:39.957906 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:39.957887    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5vxwn"
May 20 14:03:39.968145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:39.968117    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5vxwn"]
May 20 14:03:40.007728 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.007682    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq4wn\" (UniqueName: \"kubernetes.io/projected/6bccc336-51aa-4290-a0ea-71be31b86593-kube-api-access-bq4wn\") pod \"s3-init-5vxwn\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") " pod="kserve/s3-init-5vxwn"
May 20 14:03:40.007728 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.007732    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bccc336-51aa-4290-a0ea-71be31b86593-model-cache\") pod \"s3-init-5vxwn\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") " pod="kserve/s3-init-5vxwn"
May 20 14:03:40.108816 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.108779    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bq4wn\" (UniqueName: \"kubernetes.io/projected/6bccc336-51aa-4290-a0ea-71be31b86593-kube-api-access-bq4wn\") pod \"s3-init-5vxwn\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") " pod="kserve/s3-init-5vxwn"
May 20 14:03:40.108996 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.108825    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bccc336-51aa-4290-a0ea-71be31b86593-model-cache\") pod \"s3-init-5vxwn\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") " pod="kserve/s3-init-5vxwn"
May 20 14:03:40.109203 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.109186    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bccc336-51aa-4290-a0ea-71be31b86593-model-cache\") pod \"s3-init-5vxwn\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") " pod="kserve/s3-init-5vxwn"
May 20 14:03:40.117067 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.117045    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq4wn\" (UniqueName: \"kubernetes.io/projected/6bccc336-51aa-4290-a0ea-71be31b86593-kube-api-access-bq4wn\") pod \"s3-init-5vxwn\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") " pod="kserve/s3-init-5vxwn"
May 20 14:03:40.267280 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.267169    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5vxwn"
May 20 14:03:40.404956 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:40.404930    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5vxwn"]
May 20 14:03:40.406876 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:03:40.406825    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bccc336_51aa_4290_a0ea_71be31b86593.slice/crio-31d33962f84820ca9882ab1ba5aba6c55bf006e79ad041e38fb5820e0daca21a WatchSource:0}: Error finding container 31d33962f84820ca9882ab1ba5aba6c55bf006e79ad041e38fb5820e0daca21a: Status 404 returned error can't find the container with id 31d33962f84820ca9882ab1ba5aba6c55bf006e79ad041e38fb5820e0daca21a
May 20 14:03:41.171238 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:41.171197    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5vxwn" event={"ID":"6bccc336-51aa-4290-a0ea-71be31b86593","Type":"ContainerStarted","Data":"31d33962f84820ca9882ab1ba5aba6c55bf006e79ad041e38fb5820e0daca21a"}
May 20 14:03:44.186196 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:44.186160    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5vxwn" event={"ID":"6bccc336-51aa-4290-a0ea-71be31b86593","Type":"ContainerStarted","Data":"f640f0ed9e446ba35f381063a231d731bffebdb8b2a8bf442cb4f68e5c8e6767"}
May 20 14:03:46.196223 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:46.196185    2554 generic.go:358] "Generic (PLEG): container finished" podID="6bccc336-51aa-4290-a0ea-71be31b86593" containerID="f640f0ed9e446ba35f381063a231d731bffebdb8b2a8bf442cb4f68e5c8e6767" exitCode=0
May 20 14:03:46.196580 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:46.196257    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5vxwn" event={"ID":"6bccc336-51aa-4290-a0ea-71be31b86593","Type":"ContainerDied","Data":"f640f0ed9e446ba35f381063a231d731bffebdb8b2a8bf442cb4f68e5c8e6767"}
May 20 14:03:51.220360 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:51.220324    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5vxwn" event={"ID":"6bccc336-51aa-4290-a0ea-71be31b86593","Type":"ContainerStarted","Data":"1b9a073b0d18d375738dddb5d24d17e3c98cf4ca31854561305d12fbfb980e37"}
May 20 14:03:51.249628 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:03:51.249565    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-5vxwn" podStartSLOduration=1.959209547 podStartE2EDuration="12.249548657s" podCreationTimestamp="2026-05-20 14:03:39 +0000 UTC" firstStartedPulling="2026-05-20 14:03:40.408693674 +0000 UTC m=+805.031418720" lastFinishedPulling="2026-05-20 14:03:50.699032781 +0000 UTC m=+815.321757830" observedRunningTime="2026-05-20 14:03:51.246762624 +0000 UTC m=+815.869487689" watchObservedRunningTime="2026-05-20 14:03:51.249548657 +0000 UTC m=+815.872273725"
May 20 14:04:56.493492 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:56.493457    2554 generic.go:358] "Generic (PLEG): container finished" podID="6bccc336-51aa-4290-a0ea-71be31b86593" containerID="1b9a073b0d18d375738dddb5d24d17e3c98cf4ca31854561305d12fbfb980e37" exitCode=0
May 20 14:04:56.493960 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:56.493530    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5vxwn" event={"ID":"6bccc336-51aa-4290-a0ea-71be31b86593","Type":"ContainerDied","Data":"1b9a073b0d18d375738dddb5d24d17e3c98cf4ca31854561305d12fbfb980e37"}
May 20 14:04:57.647200 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:57.647172    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5vxwn"
May 20 14:04:57.830833 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:57.830724    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bccc336-51aa-4290-a0ea-71be31b86593-model-cache\") pod \"6bccc336-51aa-4290-a0ea-71be31b86593\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") "
May 20 14:04:57.831041 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:57.830903    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq4wn\" (UniqueName: \"kubernetes.io/projected/6bccc336-51aa-4290-a0ea-71be31b86593-kube-api-access-bq4wn\") pod \"6bccc336-51aa-4290-a0ea-71be31b86593\" (UID: \"6bccc336-51aa-4290-a0ea-71be31b86593\") "
May 20 14:04:57.833005 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:57.832969    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bccc336-51aa-4290-a0ea-71be31b86593-kube-api-access-bq4wn" (OuterVolumeSpecName: "kube-api-access-bq4wn") pod "6bccc336-51aa-4290-a0ea-71be31b86593" (UID: "6bccc336-51aa-4290-a0ea-71be31b86593"). InnerVolumeSpecName "kube-api-access-bq4wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:04:57.854761 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:57.854713    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bccc336-51aa-4290-a0ea-71be31b86593-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bccc336-51aa-4290-a0ea-71be31b86593" (UID: "6bccc336-51aa-4290-a0ea-71be31b86593"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:04:57.932558 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:57.932521    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq4wn\" (UniqueName: \"kubernetes.io/projected/6bccc336-51aa-4290-a0ea-71be31b86593-kube-api-access-bq4wn\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:04:57.932558 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:57.932555    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bccc336-51aa-4290-a0ea-71be31b86593-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:04:58.503810 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:58.503728    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5vxwn"
May 20 14:04:58.503810 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:58.503751    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5vxwn" event={"ID":"6bccc336-51aa-4290-a0ea-71be31b86593","Type":"ContainerDied","Data":"31d33962f84820ca9882ab1ba5aba6c55bf006e79ad041e38fb5820e0daca21a"}
May 20 14:04:58.503810 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:04:58.503778    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d33962f84820ca9882ab1ba5aba6c55bf006e79ad041e38fb5820e0daca21a"
May 20 14:05:08.901987 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.901896    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"]
May 20 14:05:08.902558 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.902538    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bccc336-51aa-4290-a0ea-71be31b86593" containerName="s3-init"
May 20 14:05:08.902640 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.902563    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bccc336-51aa-4290-a0ea-71be31b86593" containerName="s3-init"
May 20 14:05:08.902640 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.902595    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bccc336-51aa-4290-a0ea-71be31b86593" containerName="download-hf-model"
May 20 14:05:08.902640 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.902605    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bccc336-51aa-4290-a0ea-71be31b86593" containerName="download-hf-model"
May 20 14:05:08.902802 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.902718    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="6bccc336-51aa-4290-a0ea-71be31b86593" containerName="s3-init"
May 20 14:05:08.905316 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.905294    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.908078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.908031    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
May 20 14:05:08.908078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.908062    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:05:08.908402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.908310    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:05:08.908402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.908378    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-dwlz9\""
May 20 14:05:08.920349 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.920319    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"]
May 20 14:05:08.926452 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926345    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926452 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926452 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926426    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgmf\" (UniqueName: \"kubernetes.io/projected/89d1d61f-8fdc-4183-9de2-e78fe30178f3-kube-api-access-2tgmf\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926466    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926506    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926580    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:08.926915 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:08.926641    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.027764 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.027723    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.027764 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.027764    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028027 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.027787    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgmf\" (UniqueName: \"kubernetes.io/projected/89d1d61f-8fdc-4183-9de2-e78fe30178f3-kube-api-access-2tgmf\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028027 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.027929    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028027 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.027996    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028197 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028052    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028197 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028084    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028197 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028112    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028197 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028155    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028413 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028284    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-workload-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028413 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028394    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-workload-certs\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028530 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028509    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-data\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028766 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028740    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.028888 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.028833    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-credential-socket\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.030327 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.030298    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-envoy\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.030898 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.030873    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.036048 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.036023    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d1d61f-8fdc-4183-9de2-e78fe30178f3-istio-token\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.036623 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.036598    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgmf\" (UniqueName: \"kubernetes.io/projected/89d1d61f-8fdc-4183-9de2-e78fe30178f3-kube-api-access-2tgmf\") pod \"router-gateway-1-openshift-default-7b9cf87876-d6vqg\" (UID: \"89d1d61f-8fdc-4183-9de2-e78fe30178f3\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.221914 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.221875    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:09.386437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.386404    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"]
May 20 14:05:09.388011 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:05:09.387982    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d1d61f_8fdc_4183_9de2_e78fe30178f3.slice/crio-48e5c25027ac088e70fb2d601a3e4a0304f7f9075b9e5aa1bb0f947193609ccf WatchSource:0}: Error finding container 48e5c25027ac088e70fb2d601a3e4a0304f7f9075b9e5aa1bb0f947193609ccf: Status 404 returned error can't find the container with id 48e5c25027ac088e70fb2d601a3e4a0304f7f9075b9e5aa1bb0f947193609ccf
May 20 14:05:09.390203 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.390163    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:05:09.390312 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.390249    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:05:09.390312 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.390290    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:05:09.552066 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.551956    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg" event={"ID":"89d1d61f-8fdc-4183-9de2-e78fe30178f3","Type":"ContainerStarted","Data":"97d22e7873d14e270b0434b4856bab760a218216727d76241f698cae48de4fea"}
May 20 14:05:09.552066 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.552012    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg" event={"ID":"89d1d61f-8fdc-4183-9de2-e78fe30178f3","Type":"ContainerStarted","Data":"48e5c25027ac088e70fb2d601a3e4a0304f7f9075b9e5aa1bb0f947193609ccf"}
May 20 14:05:09.573523 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:09.573465    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg" podStartSLOduration=1.5734492169999998 podStartE2EDuration="1.573449217s" podCreationTimestamp="2026-05-20 14:05:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:05:09.569526493 +0000 UTC m=+894.192251563" watchObservedRunningTime="2026-05-20 14:05:09.573449217 +0000 UTC m=+894.196174322"
May 20 14:05:10.222456 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:10.222412    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:10.227478 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:10.227450    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:10.556358 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:10.556266    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:10.557437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:10.557415    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-7b9cf87876-d6vqg"
May 20 14:05:15.943991 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:15.943959    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:05:15.947004 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:15.946980    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:05:20.272699 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.272656    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"]
May 20 14:05:20.275493 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.275474    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.278051 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.278022    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:05:20.278189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.278135    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
May 20 14:05:20.278739 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.278711    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:05:20.285437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.285414    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"]
May 20 14:05:20.343004 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.342971    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmq9\" (UniqueName: \"kubernetes.io/projected/d19945e1-ab36-481a-b141-748adf5cc143-kube-api-access-lwmq9\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.343179 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.343021    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.343179 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.343148    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.343270 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.343186    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.343270 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.343260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19945e1-ab36-481a-b141-748adf5cc143-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.343360 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.343335    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.343404 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.343371    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444365 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444327    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444365 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444369    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444419    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19945e1-ab36-481a-b141-748adf5cc143-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444463    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444481    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444512    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmq9\" (UniqueName: \"kubernetes.io/projected/d19945e1-ab36-481a-b141-748adf5cc143-kube-api-access-lwmq9\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444543    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444868 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444796    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444956 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444860    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.444956 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.444913    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.445058 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.445005    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.446791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.446769    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.447101 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.447082    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19945e1-ab36-481a-b141-748adf5cc143-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.465246 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.465208    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmq9\" (UniqueName: \"kubernetes.io/projected/d19945e1-ab36-481a-b141-748adf5cc143-kube-api-access-lwmq9\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.586775 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.586674    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:05:20.735939 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:20.735914    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"]
May 20 14:05:20.737162 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:05:20.737137    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19945e1_ab36_481a_b141_748adf5cc143.slice/crio-9290d15c17ac61f1826e7dad3b483645354a97a89eead3ff758ce7d117ec6094 WatchSource:0}: Error finding container 9290d15c17ac61f1826e7dad3b483645354a97a89eead3ff758ce7d117ec6094: Status 404 returned error can't find the container with id 9290d15c17ac61f1826e7dad3b483645354a97a89eead3ff758ce7d117ec6094
May 20 14:05:21.608363 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:21.608313    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4" event={"ID":"d19945e1-ab36-481a-b141-748adf5cc143","Type":"ContainerStarted","Data":"dcc40f72073e87c89074e0a6977ddd6d53058bdd965b7d01e12b87a735d6027b"}
May 20 14:05:21.608900 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:21.608371    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4" event={"ID":"d19945e1-ab36-481a-b141-748adf5cc143","Type":"ContainerStarted","Data":"9290d15c17ac61f1826e7dad3b483645354a97a89eead3ff758ce7d117ec6094"}
May 20 14:05:27.570032 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.569983    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"]
May 20 14:05:27.575182 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.575151    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.577822 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.577792    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-h5glt\""
May 20 14:05:27.578156 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.578063    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 20 14:05:27.585140 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.585108    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"]
May 20 14:05:27.710534 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.710494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.710705 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.710544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgkt\" (UniqueName: \"kubernetes.io/projected/01cce085-9afc-473c-bb0f-ccfcae5e7952-kube-api-access-7jgkt\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.710705 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.710661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.710797 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.710721    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.710797 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.710755    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.710797 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.710778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01cce085-9afc-473c-bb0f-ccfcae5e7952-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.811884 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.811819    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812130 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.811948    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812130 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812024    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01cce085-9afc-473c-bb0f-ccfcae5e7952-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812130 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812109    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812320 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812157    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgkt\" (UniqueName: \"kubernetes.io/projected/01cce085-9afc-473c-bb0f-ccfcae5e7952-kube-api-access-7jgkt\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812320 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812186    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812320 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812273    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812476 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812410    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812529 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812505    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.812575 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.812559    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.815971 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.815897    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01cce085-9afc-473c-bb0f-ccfcae5e7952-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.824995 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.824824    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgkt\" (UniqueName: \"kubernetes.io/projected/01cce085-9afc-473c-bb0f-ccfcae5e7952-kube-api-access-7jgkt\") pod \"scheduler-inline-config-test-kserve-router-scheduler-95466xddxz\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:27.890222 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:27.890182    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:05:28.048404 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:28.048374    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"]
May 20 14:05:28.050777 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:05:28.050740    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cce085_9afc_473c_bb0f_ccfcae5e7952.slice/crio-fa6b2e657ea4f0f65c83ae5b0071baf6ed7a71687c821140097a9b38d12788e9 WatchSource:0}: Error finding container fa6b2e657ea4f0f65c83ae5b0071baf6ed7a71687c821140097a9b38d12788e9: Status 404 returned error can't find the container with id fa6b2e657ea4f0f65c83ae5b0071baf6ed7a71687c821140097a9b38d12788e9
May 20 14:05:28.641891 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:28.641825    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerStarted","Data":"7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01"}
May 20 14:05:28.641891 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:28.641896    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerStarted","Data":"fa6b2e657ea4f0f65c83ae5b0071baf6ed7a71687c821140097a9b38d12788e9"}
May 20 14:05:28.847069 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:28.847029    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"]
May 20 14:05:28.847395 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:28.847365    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4" podUID="d19945e1-ab36-481a-b141-748adf5cc143" containerName="storage-initializer" containerID="cri-o://dcc40f72073e87c89074e0a6977ddd6d53058bdd965b7d01e12b87a735d6027b" gracePeriod=30
May 20 14:05:29.647761 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:29.647720    2554 generic.go:358] "Generic (PLEG): container finished" podID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerID="7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01" exitCode=0
May 20 14:05:29.648261 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:29.647765    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerDied","Data":"7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01"}
May 20 14:05:31.659992 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:05:31.659956    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerStarted","Data":"16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d"}
May 20 14:06:00.799339 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.799304    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4_d19945e1-ab36-481a-b141-748adf5cc143/storage-initializer/0.log"
May 20 14:06:00.799733 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.799351    2554 generic.go:358] "Generic (PLEG): container finished" podID="d19945e1-ab36-481a-b141-748adf5cc143" containerID="dcc40f72073e87c89074e0a6977ddd6d53058bdd965b7d01e12b87a735d6027b" exitCode=137
May 20 14:06:00.799733 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.799429    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4" event={"ID":"d19945e1-ab36-481a-b141-748adf5cc143","Type":"ContainerDied","Data":"dcc40f72073e87c89074e0a6977ddd6d53058bdd965b7d01e12b87a735d6027b"}
May 20 14:06:00.826826 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.826800    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4_d19945e1-ab36-481a-b141-748adf5cc143/storage-initializer/0.log"
May 20 14:06:00.826980 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.826897    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:06:00.953291 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953251    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwmq9\" (UniqueName: \"kubernetes.io/projected/d19945e1-ab36-481a-b141-748adf5cc143-kube-api-access-lwmq9\") pod \"d19945e1-ab36-481a-b141-748adf5cc143\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") "
May 20 14:06:00.953462 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953299    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-model-cache\") pod \"d19945e1-ab36-481a-b141-748adf5cc143\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") "
May 20 14:06:00.953462 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953340    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-home\") pod \"d19945e1-ab36-481a-b141-748adf5cc143\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") "
May 20 14:06:00.953462 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953365    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-tmp-dir\") pod \"d19945e1-ab36-481a-b141-748adf5cc143\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") "
May 20 14:06:00.953462 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953408    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-kserve-provision-location\") pod \"d19945e1-ab36-481a-b141-748adf5cc143\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") "
May 20 14:06:00.953462 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953441    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-dshm\") pod \"d19945e1-ab36-481a-b141-748adf5cc143\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") "
May 20 14:06:00.953747 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953482    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19945e1-ab36-481a-b141-748adf5cc143-tls-certs\") pod \"d19945e1-ab36-481a-b141-748adf5cc143\" (UID: \"d19945e1-ab36-481a-b141-748adf5cc143\") "
May 20 14:06:00.953747 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953627    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-model-cache" (OuterVolumeSpecName: "model-cache") pod "d19945e1-ab36-481a-b141-748adf5cc143" (UID: "d19945e1-ab36-481a-b141-748adf5cc143"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:00.953747 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953707    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d19945e1-ab36-481a-b141-748adf5cc143" (UID: "d19945e1-ab36-481a-b141-748adf5cc143"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:00.953933 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953827    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:00.953933 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.953868    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:00.954298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.954261    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-home" (OuterVolumeSpecName: "home") pod "d19945e1-ab36-481a-b141-748adf5cc143" (UID: "d19945e1-ab36-481a-b141-748adf5cc143"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:00.956008 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.955975    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19945e1-ab36-481a-b141-748adf5cc143-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d19945e1-ab36-481a-b141-748adf5cc143" (UID: "d19945e1-ab36-481a-b141-748adf5cc143"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:06:00.956143 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.956119    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-dshm" (OuterVolumeSpecName: "dshm") pod "d19945e1-ab36-481a-b141-748adf5cc143" (UID: "d19945e1-ab36-481a-b141-748adf5cc143"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:00.956215 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:00.956196    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19945e1-ab36-481a-b141-748adf5cc143-kube-api-access-lwmq9" (OuterVolumeSpecName: "kube-api-access-lwmq9") pod "d19945e1-ab36-481a-b141-748adf5cc143" (UID: "d19945e1-ab36-481a-b141-748adf5cc143"). InnerVolumeSpecName "kube-api-access-lwmq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:06:01.013008 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.012962    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d19945e1-ab36-481a-b141-748adf5cc143" (UID: "d19945e1-ab36-481a-b141-748adf5cc143"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:01.055402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.055365    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:01.055402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.055399    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:01.055402 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.055411    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19945e1-ab36-481a-b141-748adf5cc143-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:01.055635 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.055420    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19945e1-ab36-481a-b141-748adf5cc143-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:01.055635 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.055430    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwmq9\" (UniqueName: \"kubernetes.io/projected/d19945e1-ab36-481a-b141-748adf5cc143-kube-api-access-lwmq9\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:01.805201 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.805173    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4_d19945e1-ab36-481a-b141-748adf5cc143/storage-initializer/0.log"
May 20 14:06:01.805602 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.805324    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"
May 20 14:06:01.805602 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.805323    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4" event={"ID":"d19945e1-ab36-481a-b141-748adf5cc143","Type":"ContainerDied","Data":"9290d15c17ac61f1826e7dad3b483645354a97a89eead3ff758ce7d117ec6094"}
May 20 14:06:01.805602 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.805375    2554 scope.go:117] "RemoveContainer" containerID="dcc40f72073e87c89074e0a6977ddd6d53058bdd965b7d01e12b87a735d6027b"
May 20 14:06:01.849873 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.849826    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"]
May 20 14:06:01.856361 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:01.856327    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-75f496cc6428bk4"]
May 20 14:06:02.177563 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:02.177530    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19945e1-ab36-481a-b141-748adf5cc143" path="/var/lib/kubelet/pods/d19945e1-ab36-481a-b141-748adf5cc143/volumes"
May 20 14:06:02.813249 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:02.813212    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerStarted","Data":"f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883"}
May 20 14:06:02.813710 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:02.813582    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:02.816558 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:02.816537    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:02.839473 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:02.839418    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" podStartSLOduration=3.228573447 podStartE2EDuration="35.839402285s" podCreationTimestamp="2026-05-20 14:05:27 +0000 UTC" firstStartedPulling="2026-05-20 14:05:29.64928566 +0000 UTC m=+914.272010706" lastFinishedPulling="2026-05-20 14:06:02.260114493 +0000 UTC m=+946.882839544" observedRunningTime="2026-05-20 14:06:02.835951118 +0000 UTC m=+947.458676190" watchObservedRunningTime="2026-05-20 14:06:02.839402285 +0000 UTC m=+947.462127352"
May 20 14:06:07.891240 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:07.891206    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:07.891240 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:07.891252    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:07.891888 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:07.891461    2554 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="tokenizer" probeResult="failure" output="Get \"http://10.132.0.58:8082/healthz\": dial tcp 10.132.0.58:8082: connect: connection refused"
May 20 14:06:17.892147 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:17.892110    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:17.893438 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:17.893418    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:19.502336 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:19.502301    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"]
May 20 14:06:19.885078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:19.884954    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="main" containerID="cri-o://16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d" gracePeriod=30
May 20 14:06:19.885078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:19.885019    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="tokenizer" containerID="cri-o://f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883" gracePeriod=30
May 20 14:06:20.891206 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:20.891174    2554 generic.go:358] "Generic (PLEG): container finished" podID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerID="16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d" exitCode=0
May 20 14:06:20.891593 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:20.891250    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerDied","Data":"16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d"}
May 20 14:06:21.258835 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.258804    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:21.449510 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449474    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01cce085-9afc-473c-bb0f-ccfcae5e7952-tls-certs\") pod \"01cce085-9afc-473c-bb0f-ccfcae5e7952\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") "
May 20 14:06:21.449510 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449514    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jgkt\" (UniqueName: \"kubernetes.io/projected/01cce085-9afc-473c-bb0f-ccfcae5e7952-kube-api-access-7jgkt\") pod \"01cce085-9afc-473c-bb0f-ccfcae5e7952\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") "
May 20 14:06:21.449770 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449532    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-cache\") pod \"01cce085-9afc-473c-bb0f-ccfcae5e7952\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") "
May 20 14:06:21.449770 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449553    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-uds\") pod \"01cce085-9afc-473c-bb0f-ccfcae5e7952\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") "
May 20 14:06:21.449770 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449601    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-tmp\") pod \"01cce085-9afc-473c-bb0f-ccfcae5e7952\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") "
May 20 14:06:21.449770 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449622    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-kserve-provision-location\") pod \"01cce085-9afc-473c-bb0f-ccfcae5e7952\" (UID: \"01cce085-9afc-473c-bb0f-ccfcae5e7952\") "
May 20 14:06:21.449989 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449816    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "01cce085-9afc-473c-bb0f-ccfcae5e7952" (UID: "01cce085-9afc-473c-bb0f-ccfcae5e7952"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:21.449989 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449833    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "01cce085-9afc-473c-bb0f-ccfcae5e7952" (UID: "01cce085-9afc-473c-bb0f-ccfcae5e7952"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:21.449989 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.449981    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "01cce085-9afc-473c-bb0f-ccfcae5e7952" (UID: "01cce085-9afc-473c-bb0f-ccfcae5e7952"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:21.450345 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.450323    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01cce085-9afc-473c-bb0f-ccfcae5e7952" (UID: "01cce085-9afc-473c-bb0f-ccfcae5e7952"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:21.451741 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.451715    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cce085-9afc-473c-bb0f-ccfcae5e7952-kube-api-access-7jgkt" (OuterVolumeSpecName: "kube-api-access-7jgkt") pod "01cce085-9afc-473c-bb0f-ccfcae5e7952" (UID: "01cce085-9afc-473c-bb0f-ccfcae5e7952"). InnerVolumeSpecName "kube-api-access-7jgkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:06:21.451866 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.451721    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01cce085-9afc-473c-bb0f-ccfcae5e7952-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01cce085-9afc-473c-bb0f-ccfcae5e7952" (UID: "01cce085-9afc-473c-bb0f-ccfcae5e7952"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:06:21.551351 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.551311    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01cce085-9afc-473c-bb0f-ccfcae5e7952-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:21.551351 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.551352    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgkt\" (UniqueName: \"kubernetes.io/projected/01cce085-9afc-473c-bb0f-ccfcae5e7952-kube-api-access-7jgkt\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:21.551555 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.551369    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:21.551555 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.551385    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:21.551555 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.551400    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:21.551555 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.551414    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01cce085-9afc-473c-bb0f-ccfcae5e7952-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:06:21.898320 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.898228    2554 generic.go:358] "Generic (PLEG): container finished" podID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerID="f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883" exitCode=0
May 20 14:06:21.898666 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.898314    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"
May 20 14:06:21.898666 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.898313    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerDied","Data":"f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883"}
May 20 14:06:21.898666 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.898454    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz" event={"ID":"01cce085-9afc-473c-bb0f-ccfcae5e7952","Type":"ContainerDied","Data":"fa6b2e657ea4f0f65c83ae5b0071baf6ed7a71687c821140097a9b38d12788e9"}
May 20 14:06:21.898666 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.898477    2554 scope.go:117] "RemoveContainer" containerID="f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883"
May 20 14:06:21.909749 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.909679    2554 scope.go:117] "RemoveContainer" containerID="16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d"
May 20 14:06:21.920311 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.920285    2554 scope.go:117] "RemoveContainer" containerID="7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01"
May 20 14:06:21.927424 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.927394    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"]
May 20 14:06:21.929621 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.929596    2554 scope.go:117] "RemoveContainer" containerID="f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883"
May 20 14:06:21.929939 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:06:21.929919    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883\": container with ID starting with f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883 not found: ID does not exist" containerID="f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883"
May 20 14:06:21.930005 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.929950    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883"} err="failed to get container status \"f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883\": rpc error: code = NotFound desc = could not find container \"f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883\": container with ID starting with f3a5ab835a35da028339d16327ae77f3370c4164093661955af97fda04d2e883 not found: ID does not exist"
May 20 14:06:21.930005 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.929974    2554 scope.go:117] "RemoveContainer" containerID="16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d"
May 20 14:06:21.930280 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:06:21.930257    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d\": container with ID starting with 16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d not found: ID does not exist" containerID="16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d"
May 20 14:06:21.930327 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.930290    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d"} err="failed to get container status \"16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d\": rpc error: code = NotFound desc = could not find container \"16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d\": container with ID starting with 16d6390f15036c7e50a747f146704b11ab84cccaaf6ae647402805448059e35d not found: ID does not exist"
May 20 14:06:21.930327 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.930314    2554 scope.go:117] "RemoveContainer" containerID="7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01"
May 20 14:06:21.930593 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:06:21.930573    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01\": container with ID starting with 7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01 not found: ID does not exist" containerID="7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01"
May 20 14:06:21.930652 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.930597    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01"} err="failed to get container status \"7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01\": rpc error: code = NotFound desc = could not find container \"7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01\": container with ID starting with 7c6ddd40e3efb41194d74c4a171a4d726880114e5478df2d1a7d72b67c857d01 not found: ID does not exist"
May 20 14:06:21.932101 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:21.932081    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-95466xddxz"]
May 20 14:06:22.176513 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:22.176477    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" path="/var/lib/kubelet/pods/01cce085-9afc-473c-bb0f-ccfcae5e7952/volumes"
May 20 14:06:43.363825 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.363745    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"]
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364240    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="main"
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364252    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="main"
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364261    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="tokenizer"
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364266    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="tokenizer"
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364274    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19945e1-ab36-481a-b141-748adf5cc143" containerName="storage-initializer"
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364280    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19945e1-ab36-481a-b141-748adf5cc143" containerName="storage-initializer"
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364286    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="storage-initializer"
May 20 14:06:43.364298 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364291    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="storage-initializer"
May 20 14:06:43.364543 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364347    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="d19945e1-ab36-481a-b141-748adf5cc143" containerName="storage-initializer"
May 20 14:06:43.364543 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364356    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="main"
May 20 14:06:43.364543 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.364364    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="01cce085-9afc-473c-bb0f-ccfcae5e7952" containerName="tokenizer"
May 20 14:06:43.366707 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.366689    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.369192 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.369173    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:06:43.370029 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.370010    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:06:43.370129 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.370012    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 20 14:06:43.370129 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.370012    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-4hq7q\""
May 20 14:06:43.376223 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.376197    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"]
May 20 14:06:43.451951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.451914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe970d9-c4db-4550-ad84-ff27d4fae632-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.451951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.451954    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.452189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.452028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7clw\" (UniqueName: \"kubernetes.io/projected/dbe970d9-c4db-4550-ad84-ff27d4fae632-kube-api-access-x7clw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.452189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.452091    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.452189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.452137    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.452295 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.452236    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553128 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553093    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe970d9-c4db-4550-ad84-ff27d4fae632-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553128 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553133    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553348 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553256    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x7clw\" (UniqueName: \"kubernetes.io/projected/dbe970d9-c4db-4550-ad84-ff27d4fae632-kube-api-access-x7clw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553348 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553298    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553348 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553343    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553489 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553376    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553706 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553679    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553863 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553746    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553863 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553794    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.553863 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.553820    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.555851 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.555813    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe970d9-c4db-4550-ad84-ff27d4fae632-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.562021 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.561993    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7clw\" (UniqueName: \"kubernetes.io/projected/dbe970d9-c4db-4550-ad84-ff27d4fae632-kube-api-access-x7clw\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.677152 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.677106    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:43.824091 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.824049    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"]
May 20 14:06:43.824604 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:06:43.824581    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe970d9_c4db_4550_ad84_ff27d4fae632.slice/crio-87ea731be38846a048c0726e0c751277c80fa0ee6ccdde223a2f0a0e31408ec2 WatchSource:0}: Error finding container 87ea731be38846a048c0726e0c751277c80fa0ee6ccdde223a2f0a0e31408ec2: Status 404 returned error can't find the container with id 87ea731be38846a048c0726e0c751277c80fa0ee6ccdde223a2f0a0e31408ec2
May 20 14:06:43.826834 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.826814    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:06:43.991035 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.990950    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerStarted","Data":"ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642"}
May 20 14:06:43.991035 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:43.990995    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerStarted","Data":"87ea731be38846a048c0726e0c751277c80fa0ee6ccdde223a2f0a0e31408ec2"}
May 20 14:06:44.997110 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:44.997067    2554 generic.go:358] "Generic (PLEG): container finished" podID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerID="ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642" exitCode=0
May 20 14:06:44.997583 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:44.997166    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerDied","Data":"ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642"}
May 20 14:06:46.003295 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:46.003261    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerStarted","Data":"383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13"}
May 20 14:06:46.003748 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:46.003308    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerStarted","Data":"3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275"}
May 20 14:06:46.003748 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:46.003373    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:46.028000 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:46.027948    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" podStartSLOduration=3.027931172 podStartE2EDuration="3.027931172s" podCreationTimestamp="2026-05-20 14:06:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:06:46.022618634 +0000 UTC m=+990.645343707" watchObservedRunningTime="2026-05-20 14:06:46.027931172 +0000 UTC m=+990.650656240"
May 20 14:06:53.677607 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:53.677557    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:53.677607 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:53.677613    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:53.680277 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:53.680244    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:06:54.040003 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:06:54.039923    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:07:15.045554 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:15.045522    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:07:15.977647 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:15.977579    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"]
May 20 14:07:15.978547 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:15.978510    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="main" containerID="cri-o://3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275" gracePeriod=30
May 20 14:07:15.979277 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:15.978796    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="tokenizer" containerID="cri-o://383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13" gracePeriod=30
May 20 14:07:16.129992 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:16.129947    2554 generic.go:358] "Generic (PLEG): container finished" podID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerID="3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275" exitCode=0
May 20 14:07:16.130359 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:16.130022    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerDied","Data":"3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275"}
May 20 14:07:17.347610 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.347582    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:07:17.367266 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367231    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7clw\" (UniqueName: \"kubernetes.io/projected/dbe970d9-c4db-4550-ad84-ff27d4fae632-kube-api-access-x7clw\") pod \"dbe970d9-c4db-4550-ad84-ff27d4fae632\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") "
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367302    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-cache\") pod \"dbe970d9-c4db-4550-ad84-ff27d4fae632\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") "
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367352    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-kserve-provision-location\") pod \"dbe970d9-c4db-4550-ad84-ff27d4fae632\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") "
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367392    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe970d9-c4db-4550-ad84-ff27d4fae632-tls-certs\") pod \"dbe970d9-c4db-4550-ad84-ff27d4fae632\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") "
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367439    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-uds\") pod \"dbe970d9-c4db-4550-ad84-ff27d4fae632\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") "
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367533    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-tmp\") pod \"dbe970d9-c4db-4550-ad84-ff27d4fae632\" (UID: \"dbe970d9-c4db-4550-ad84-ff27d4fae632\") "
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367570    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "dbe970d9-c4db-4550-ad84-ff27d4fae632" (UID: "dbe970d9-c4db-4550-ad84-ff27d4fae632"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367749    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "dbe970d9-c4db-4550-ad84-ff27d4fae632" (UID: "dbe970d9-c4db-4550-ad84-ff27d4fae632"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367860    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.367881    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:07:17.368184 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.368089    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "dbe970d9-c4db-4550-ad84-ff27d4fae632" (UID: "dbe970d9-c4db-4550-ad84-ff27d4fae632"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:17.368888 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.368610    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbe970d9-c4db-4550-ad84-ff27d4fae632" (UID: "dbe970d9-c4db-4550-ad84-ff27d4fae632"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:17.370031 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.370007    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe970d9-c4db-4550-ad84-ff27d4fae632-kube-api-access-x7clw" (OuterVolumeSpecName: "kube-api-access-x7clw") pod "dbe970d9-c4db-4550-ad84-ff27d4fae632" (UID: "dbe970d9-c4db-4550-ad84-ff27d4fae632"). InnerVolumeSpecName "kube-api-access-x7clw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:07:17.371258 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.371231    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe970d9-c4db-4550-ad84-ff27d4fae632-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbe970d9-c4db-4550-ad84-ff27d4fae632" (UID: "dbe970d9-c4db-4550-ad84-ff27d4fae632"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:07:17.468329 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.468286    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7clw\" (UniqueName: \"kubernetes.io/projected/dbe970d9-c4db-4550-ad84-ff27d4fae632-kube-api-access-x7clw\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:07:17.468329 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.468319    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:07:17.468329 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.468334    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbe970d9-c4db-4550-ad84-ff27d4fae632-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:07:17.468579 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:17.468349    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe970d9-c4db-4550-ad84-ff27d4fae632-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:07:18.150443 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.150404    2554 generic.go:358] "Generic (PLEG): container finished" podID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerID="383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13" exitCode=0
May 20 14:07:18.150625 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.150480    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerDied","Data":"383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13"}
May 20 14:07:18.150625 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.150506    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"
May 20 14:07:18.150625 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.150521    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg" event={"ID":"dbe970d9-c4db-4550-ad84-ff27d4fae632","Type":"ContainerDied","Data":"87ea731be38846a048c0726e0c751277c80fa0ee6ccdde223a2f0a0e31408ec2"}
May 20 14:07:18.150625 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.150538    2554 scope.go:117] "RemoveContainer" containerID="383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13"
May 20 14:07:18.159930 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.159905    2554 scope.go:117] "RemoveContainer" containerID="3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275"
May 20 14:07:18.168140 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.168110    2554 scope.go:117] "RemoveContainer" containerID="ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642"
May 20 14:07:18.178601 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.178575    2554 scope.go:117] "RemoveContainer" containerID="383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13"
May 20 14:07:18.178952 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:07:18.178925    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13\": container with ID starting with 383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13 not found: ID does not exist" containerID="383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13"
May 20 14:07:18.179034 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.178964    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13"} err="failed to get container status \"383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13\": rpc error: code = NotFound desc = could not find container \"383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13\": container with ID starting with 383126cb3d55f5810ca40906b51fd346a24d4935d63b6cb40012e09b5deb9d13 not found: ID does not exist"
May 20 14:07:18.179034 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.178990    2554 scope.go:117] "RemoveContainer" containerID="3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275"
May 20 14:07:18.179255 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:07:18.179235    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275\": container with ID starting with 3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275 not found: ID does not exist" containerID="3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275"
May 20 14:07:18.179324 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.179265    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275"} err="failed to get container status \"3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275\": rpc error: code = NotFound desc = could not find container \"3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275\": container with ID starting with 3b87221478ed30dec2fbea4d4f5e5106605bd1b407d8ca90ba582c1c062e6275 not found: ID does not exist"
May 20 14:07:18.179324 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.179291    2554 scope.go:117] "RemoveContainer" containerID="ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642"
May 20 14:07:18.179553 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:07:18.179535    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642\": container with ID starting with ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642 not found: ID does not exist" containerID="ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642"
May 20 14:07:18.179595 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.179558    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642"} err="failed to get container status \"ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642\": rpc error: code = NotFound desc = could not find container \"ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642\": container with ID starting with ef1e7bf675d27d7c458729b43e67e4a42cb74d5076b7c7f23b96bd5b3f408642 not found: ID does not exist"
May 20 14:07:18.179715 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.179699    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"]
May 20 14:07:18.179755 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:18.179727    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-579b56x6tg"]
May 20 14:07:20.177788 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.177751    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" path="/var/lib/kubelet/pods/dbe970d9-c4db-4550-ad84-ff27d4fae632/volumes"
May 20 14:07:20.986857 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.986807    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"]
May 20 14:07:20.987363 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987341    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="tokenizer"
May 20 14:07:20.987363 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987364    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="tokenizer"
May 20 14:07:20.987531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987414    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="storage-initializer"
May 20 14:07:20.987531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987425    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="storage-initializer"
May 20 14:07:20.987531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987439    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="main"
May 20 14:07:20.987531 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987448    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="main"
May 20 14:07:20.987743 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987562    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="tokenizer"
May 20 14:07:20.987743 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.987579    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbe970d9-c4db-4550-ad84-ff27d4fae632" containerName="main"
May 20 14:07:20.991499 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.991472    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:20.994181 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.994153    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:07:20.994181 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.994181    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-l4hpq\""
May 20 14:07:20.994354 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.994158    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:07:20.994354 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:20.994275    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 20 14:07:21.003581 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.003551    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"]
May 20 14:07:21.102775 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.102736    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfphr\" (UniqueName: \"kubernetes.io/projected/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kube-api-access-zfphr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.102967 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.102802    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.102967 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.102870    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.103062 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.102964    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.103062 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.103006    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.103062 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.103030    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.204437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204054    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zfphr\" (UniqueName: \"kubernetes.io/projected/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kube-api-access-zfphr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.204437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204130    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.204437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204194    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.204437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204281    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.204437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204319    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.204437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204350    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.205185 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204528    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.205185 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204542    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.205185 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204721    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.205185 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.204768    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.207212 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.207186    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.213875 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.213827    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfphr\" (UniqueName: \"kubernetes.io/projected/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kube-api-access-zfphr\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.304146 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.304066    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:21.454695 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:21.454665    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"]
May 20 14:07:21.455986 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:07:21.455959    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c69cc14_b98c_4a7b_8cb4_786a65f9c355.slice/crio-6a9b648fa4d838df7cd3b9b8e30abe7db2cefa3ef13c0cf352e460ecf25ac62b WatchSource:0}: Error finding container 6a9b648fa4d838df7cd3b9b8e30abe7db2cefa3ef13c0cf352e460ecf25ac62b: Status 404 returned error can't find the container with id 6a9b648fa4d838df7cd3b9b8e30abe7db2cefa3ef13c0cf352e460ecf25ac62b
May 20 14:07:22.170578 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:22.170533    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerStarted","Data":"15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb"}
May 20 14:07:22.170761 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:22.170584    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerStarted","Data":"6a9b648fa4d838df7cd3b9b8e30abe7db2cefa3ef13c0cf352e460ecf25ac62b"}
May 20 14:07:23.176520 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:23.176485    2554 generic.go:358] "Generic (PLEG): container finished" podID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerID="15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb" exitCode=0
May 20 14:07:23.176969 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:23.176581    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerDied","Data":"15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb"}
May 20 14:07:24.182489 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:24.182449    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerStarted","Data":"b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329"}
May 20 14:07:24.182489 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:24.182491    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerStarted","Data":"032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349"}
May 20 14:07:24.182935 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:24.182612    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:24.206390 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:24.206326    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" podStartSLOduration=4.206311153 podStartE2EDuration="4.206311153s" podCreationTimestamp="2026-05-20 14:07:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:07:24.202914925 +0000 UTC m=+1028.825639994" watchObservedRunningTime="2026-05-20 14:07:24.206311153 +0000 UTC m=+1028.829036221"
May 20 14:07:31.305182 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:31.305148    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:31.305182 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:31.305189    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:31.308011 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:31.307982    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:32.220794 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:32.220765    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:07:53.224878 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:07:53.224826    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:08:01.135484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.135439    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"]
May 20 14:08:01.138756 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.138718    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.141493 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.141460    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 20 14:08:01.141630 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.141512    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-s5zxf\""
May 20 14:08:01.153733 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.153706    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"]
May 20 14:08:01.182603 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.182565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.182603 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.182601    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5tbx\" (UniqueName: \"kubernetes.io/projected/c9e40d11-a7b9-409a-8678-f88f805c586a-kube-api-access-r5tbx\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.182875 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.182671    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.182875 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.182746    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.182875 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.182778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.182875 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.182862    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e40d11-a7b9-409a-8678-f88f805c586a-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.283959 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.283922    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284000    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284032    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284079    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e40d11-a7b9-409a-8678-f88f805c586a-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284124    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284351 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284151    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r5tbx\" (UniqueName: \"kubernetes.io/projected/c9e40d11-a7b9-409a-8678-f88f805c586a-kube-api-access-r5tbx\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284513 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284488    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284572 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284502    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284572 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284558    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.284670 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.284608    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.286651 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.286625    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e40d11-a7b9-409a-8678-f88f805c586a-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.292341 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.292316    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5tbx\" (UniqueName: \"kubernetes.io/projected/c9e40d11-a7b9-409a-8678-f88f805c586a-kube-api-access-r5tbx\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.449351 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.449311    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:01.603505 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:01.603473    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"]
May 20 14:08:01.605443 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:08:01.605412    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e40d11_a7b9_409a_8678_f88f805c586a.slice/crio-5e9b5ba04b37a028de425fc99d9684cd020b2a21c91cb322d590fd305b4d2bf0 WatchSource:0}: Error finding container 5e9b5ba04b37a028de425fc99d9684cd020b2a21c91cb322d590fd305b4d2bf0: Status 404 returned error can't find the container with id 5e9b5ba04b37a028de425fc99d9684cd020b2a21c91cb322d590fd305b4d2bf0
May 20 14:08:02.346615 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:02.346561    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerStarted","Data":"a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae"}
May 20 14:08:02.346615 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:02.346621    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerStarted","Data":"5e9b5ba04b37a028de425fc99d9684cd020b2a21c91cb322d590fd305b4d2bf0"}
May 20 14:08:03.352791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:03.352753    2554 generic.go:358] "Generic (PLEG): container finished" podID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerID="a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae" exitCode=0
May 20 14:08:03.353234 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:03.352875    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerDied","Data":"a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae"}
May 20 14:08:04.360329 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:04.360293    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerStarted","Data":"c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c"}
May 20 14:08:04.360329 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:04.360332    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerStarted","Data":"644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0"}
May 20 14:08:04.360729 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:04.360439    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:04.385306 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:04.385246    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" podStartSLOduration=3.385229686 podStartE2EDuration="3.385229686s" podCreationTimestamp="2026-05-20 14:08:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:08:04.380764487 +0000 UTC m=+1069.003489557" watchObservedRunningTime="2026-05-20 14:08:04.385229686 +0000 UTC m=+1069.007954755"
May 20 14:08:11.450066 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:11.449980    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:11.450066 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:11.450033    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:11.453015 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:11.452987    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:12.396470 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:12.396431    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:08:22.815671 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:22.815632    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"]
May 20 14:08:22.816213 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:22.815989    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="main" containerID="cri-o://032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349" gracePeriod=30
May 20 14:08:22.816213 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:22.816024    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="tokenizer" containerID="cri-o://b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329" gracePeriod=30
May 20 14:08:23.223712 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:08:23.223680    2554 logging.go:55] [core] [Channel #74 SubChannel #75]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.60:9003", ServerName: "10.132.0.60:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.60:9003: connect: connection refused"
May 20 14:08:23.443367 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:23.443329    2554 generic.go:358] "Generic (PLEG): container finished" podID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerID="032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349" exitCode=0
May 20 14:08:23.443580 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:23.443416    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerDied","Data":"032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349"}
May 20 14:08:24.175317 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.175282    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:08:24.224422 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.224321    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.60:9003\" within 1s: context deadline exceeded"
May 20 14:08:24.306117 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306075    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-uds\") pod \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") "
May 20 14:08:24.306307 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306142    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-tmp\") pod \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") "
May 20 14:08:24.306307 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306212    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-cache\") pod \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") "
May 20 14:08:24.306307 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306299    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfphr\" (UniqueName: \"kubernetes.io/projected/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kube-api-access-zfphr\") pod \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") "
May 20 14:08:24.306491 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306354    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tls-certs\") pod \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") "
May 20 14:08:24.306491 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306397    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kserve-provision-location\") pod \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\" (UID: \"1c69cc14-b98c-4a7b-8cb4-786a65f9c355\") "
May 20 14:08:24.306491 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306405    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1c69cc14-b98c-4a7b-8cb4-786a65f9c355" (UID: "1c69cc14-b98c-4a7b-8cb4-786a65f9c355"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.306491 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306415    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "1c69cc14-b98c-4a7b-8cb4-786a65f9c355" (UID: "1c69cc14-b98c-4a7b-8cb4-786a65f9c355"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.306645 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306506    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "1c69cc14-b98c-4a7b-8cb4-786a65f9c355" (UID: "1c69cc14-b98c-4a7b-8cb4-786a65f9c355"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.306773 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306754    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:08:24.306854 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306778    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:08:24.306854 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.306793    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:08:24.307210 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.307189    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c69cc14-b98c-4a7b-8cb4-786a65f9c355" (UID: "1c69cc14-b98c-4a7b-8cb4-786a65f9c355"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:24.308532 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.308507    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c69cc14-b98c-4a7b-8cb4-786a65f9c355" (UID: "1c69cc14-b98c-4a7b-8cb4-786a65f9c355"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:08:24.308636 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.308574    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kube-api-access-zfphr" (OuterVolumeSpecName: "kube-api-access-zfphr") pod "1c69cc14-b98c-4a7b-8cb4-786a65f9c355" (UID: "1c69cc14-b98c-4a7b-8cb4-786a65f9c355"). InnerVolumeSpecName "kube-api-access-zfphr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:08:24.407603 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.407567    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfphr\" (UniqueName: \"kubernetes.io/projected/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kube-api-access-zfphr\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:08:24.407603 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.407598    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:08:24.407603 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.407609    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c69cc14-b98c-4a7b-8cb4-786a65f9c355-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:08:24.450228 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.450192    2554 generic.go:358] "Generic (PLEG): container finished" podID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerID="b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329" exitCode=0
May 20 14:08:24.450403 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.450285    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerDied","Data":"b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329"}
May 20 14:08:24.450403 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.450304    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"
May 20 14:08:24.450403 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.450335    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm" event={"ID":"1c69cc14-b98c-4a7b-8cb4-786a65f9c355","Type":"ContainerDied","Data":"6a9b648fa4d838df7cd3b9b8e30abe7db2cefa3ef13c0cf352e460ecf25ac62b"}
May 20 14:08:24.450403 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.450357    2554 scope.go:117] "RemoveContainer" containerID="b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329"
May 20 14:08:24.461008 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.460988    2554 scope.go:117] "RemoveContainer" containerID="032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349"
May 20 14:08:24.469408 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.469388    2554 scope.go:117] "RemoveContainer" containerID="15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb"
May 20 14:08:24.476736 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.476678    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"]
May 20 14:08:24.482923 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.482899    2554 scope.go:117] "RemoveContainer" containerID="b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329"
May 20 14:08:24.483326 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:08:24.483298    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329\": container with ID starting with b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329 not found: ID does not exist" containerID="b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329"
May 20 14:08:24.483417 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.483339    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329"} err="failed to get container status \"b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329\": rpc error: code = NotFound desc = could not find container \"b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329\": container with ID starting with b69433a24ec5570726f84db256c0759b50c37c18ab062082d1e40e5df1ac3329 not found: ID does not exist"
May 20 14:08:24.483417 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.483367    2554 scope.go:117] "RemoveContainer" containerID="032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349"
May 20 14:08:24.483561 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.483542    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6cb95b8m4xzm"]
May 20 14:08:24.483679 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:08:24.483644    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349\": container with ID starting with 032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349 not found: ID does not exist" containerID="032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349"
May 20 14:08:24.483679 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.483672    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349"} err="failed to get container status \"032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349\": rpc error: code = NotFound desc = could not find container \"032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349\": container with ID starting with 032f0e39ac4e05d21728d4fa07177e390802f1d6ff6c9f96ee9f4f5f3c586349 not found: ID does not exist"
May 20 14:08:24.483802 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.483688    2554 scope.go:117] "RemoveContainer" containerID="15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb"
May 20 14:08:24.483947 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:08:24.483925    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb\": container with ID starting with 15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb not found: ID does not exist" containerID="15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb"
May 20 14:08:24.484009 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:24.483945    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb"} err="failed to get container status \"15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb\": rpc error: code = NotFound desc = could not find container \"15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb\": container with ID starting with 15239ac3870004355166dd3f63af05fcbebb5c77c33dab8b28d057e9f85ee8bb not found: ID does not exist"
May 20 14:08:26.178271 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:26.178234    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" path="/var/lib/kubelet/pods/1c69cc14-b98c-4a7b-8cb4-786a65f9c355/volumes"
May 20 14:08:33.401141 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:08:33.401109    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:10:15.978066 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:15.978032    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:10:15.982933 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:15.982911    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:10:37.459707 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:37.459661    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"]
May 20 14:10:37.460342 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:37.460020    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="main" containerID="cri-o://644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0" gracePeriod=30
May 20 14:10:37.460342 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:37.460056    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="tokenizer" containerID="cri-o://c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c" gracePeriod=30
May 20 14:10:38.013597 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.013563    2554 generic.go:358] "Generic (PLEG): container finished" podID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerID="644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0" exitCode=0
May 20 14:10:38.013767 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.013638    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerDied","Data":"644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0"}
May 20 14:10:38.828690 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.828665    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:10:38.875514 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875426    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-cache\") pod \"c9e40d11-a7b9-409a-8678-f88f805c586a\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") "
May 20 14:10:38.875663 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875521    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-tmp\") pod \"c9e40d11-a7b9-409a-8678-f88f805c586a\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") "
May 20 14:10:38.875663 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875551    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5tbx\" (UniqueName: \"kubernetes.io/projected/c9e40d11-a7b9-409a-8678-f88f805c586a-kube-api-access-r5tbx\") pod \"c9e40d11-a7b9-409a-8678-f88f805c586a\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") "
May 20 14:10:38.875663 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875598    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e40d11-a7b9-409a-8678-f88f805c586a-tls-certs\") pod \"c9e40d11-a7b9-409a-8678-f88f805c586a\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") "
May 20 14:10:38.875663 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875626    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-kserve-provision-location\") pod \"c9e40d11-a7b9-409a-8678-f88f805c586a\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") "
May 20 14:10:38.875877 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875664    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-uds\") pod \"c9e40d11-a7b9-409a-8678-f88f805c586a\" (UID: \"c9e40d11-a7b9-409a-8678-f88f805c586a\") "
May 20 14:10:38.875877 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875742    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "c9e40d11-a7b9-409a-8678-f88f805c586a" (UID: "c9e40d11-a7b9-409a-8678-f88f805c586a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:38.876002 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.875927    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "c9e40d11-a7b9-409a-8678-f88f805c586a" (UID: "c9e40d11-a7b9-409a-8678-f88f805c586a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:38.876061 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.876023    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:10:38.876061 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.876037    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:10:38.876160 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.876052    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c9e40d11-a7b9-409a-8678-f88f805c586a" (UID: "c9e40d11-a7b9-409a-8678-f88f805c586a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:38.876399 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.876376    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9e40d11-a7b9-409a-8678-f88f805c586a" (UID: "c9e40d11-a7b9-409a-8678-f88f805c586a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:38.877783 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.877756    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e40d11-a7b9-409a-8678-f88f805c586a-kube-api-access-r5tbx" (OuterVolumeSpecName: "kube-api-access-r5tbx") pod "c9e40d11-a7b9-409a-8678-f88f805c586a" (UID: "c9e40d11-a7b9-409a-8678-f88f805c586a"). InnerVolumeSpecName "kube-api-access-r5tbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:10:38.877783 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.877774    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e40d11-a7b9-409a-8678-f88f805c586a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9e40d11-a7b9-409a-8678-f88f805c586a" (UID: "c9e40d11-a7b9-409a-8678-f88f805c586a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:10:38.976685 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.976645    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e40d11-a7b9-409a-8678-f88f805c586a-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:10:38.976685 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.976681    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:10:38.976685 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.976692    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9e40d11-a7b9-409a-8678-f88f805c586a-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:10:38.976967 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:38.976702    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5tbx\" (UniqueName: \"kubernetes.io/projected/c9e40d11-a7b9-409a-8678-f88f805c586a-kube-api-access-r5tbx\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:10:39.019886 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.019818    2554 generic.go:358] "Generic (PLEG): container finished" podID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerID="c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c" exitCode=0
May 20 14:10:39.020049 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.019879    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerDied","Data":"c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c"}
May 20 14:10:39.020049 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.019926    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc" event={"ID":"c9e40d11-a7b9-409a-8678-f88f805c586a","Type":"ContainerDied","Data":"5e9b5ba04b37a028de425fc99d9684cd020b2a21c91cb322d590fd305b4d2bf0"}
May 20 14:10:39.020049 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.019943    2554 scope.go:117] "RemoveContainer" containerID="c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c"
May 20 14:10:39.020049 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.019941    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"
May 20 14:10:39.029477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.029457    2554 scope.go:117] "RemoveContainer" containerID="644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0"
May 20 14:10:39.038251 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.038229    2554 scope.go:117] "RemoveContainer" containerID="a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae"
May 20 14:10:39.042909 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.042884    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"]
May 20 14:10:39.047532 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.047509    2554 scope.go:117] "RemoveContainer" containerID="c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c"
May 20 14:10:39.047888 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:10:39.047855    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c\": container with ID starting with c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c not found: ID does not exist" containerID="c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c"
May 20 14:10:39.047985 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.047899    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c"} err="failed to get container status \"c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c\": rpc error: code = NotFound desc = could not find container \"c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c\": container with ID starting with c63a2bdea84d23d5c5c5fca4a2e4a6cb8aad6d1ef4a969bd8dafc7fc8f8fb91c not found: ID does not exist"
May 20 14:10:39.047985 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.047920    2554 scope.go:117] "RemoveContainer" containerID="644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0"
May 20 14:10:39.048208 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:10:39.048190    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0\": container with ID starting with 644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0 not found: ID does not exist" containerID="644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0"
May 20 14:10:39.048269 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.048212    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0"} err="failed to get container status \"644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0\": rpc error: code = NotFound desc = could not find container \"644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0\": container with ID starting with 644ab9fbfa90501aeeb54a1426d9b1492e6235b3602029e4f3f71a83975144c0 not found: ID does not exist"
May 20 14:10:39.048269 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.048225    2554 scope.go:117] "RemoveContainer" containerID="a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae"
May 20 14:10:39.048476 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:10:39.048453    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae\": container with ID starting with a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae not found: ID does not exist" containerID="a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae"
May 20 14:10:39.048525 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.048483    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae"} err="failed to get container status \"a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae\": rpc error: code = NotFound desc = could not find container \"a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae\": container with ID starting with a0f28d8074530fefd7c72d2f25418392be7f69f4ef49a0c5a1b7bc104c83d0ae not found: ID does not exist"
May 20 14:10:39.049326 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:39.049307    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schedx5gc"]
May 20 14:10:40.177581 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:40.177543    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" path="/var/lib/kubelet/pods/c9e40d11-a7b9-409a-8678-f88f805c586a/volumes"
May 20 14:10:52.292359 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.292323    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"]
May 20 14:10:52.292983 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.292926    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="storage-initializer"
May 20 14:10:52.292983 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.292948    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="storage-initializer"
May 20 14:10:52.292983 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.292967    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="tokenizer"
May 20 14:10:52.292983 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.292974    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="tokenizer"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293000    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="main"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293010    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="main"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293030    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="tokenizer"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293039    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="tokenizer"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293061    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="storage-initializer"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293069    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="storage-initializer"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293080    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="main"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293087    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="main"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293185    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="main"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293200    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="tokenizer"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293210    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="1c69cc14-b98c-4a7b-8cb4-786a65f9c355" containerName="tokenizer"
May 20 14:10:52.293253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.293221    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="c9e40d11-a7b9-409a-8678-f88f805c586a" containerName="main"
May 20 14:10:52.296765 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.296745    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.299273 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.299252    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:10:52.299405 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.299307    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 20 14:10:52.299479 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.299253    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:10:52.299543 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.299474    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-bdhxd\""
May 20 14:10:52.308148 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.308124    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"]
May 20 14:10:52.400913 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.400879    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.401100 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.400919    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.401100 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.401019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000dab78-2620-41a2-a74c-e58809e8b312-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.401100 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.401069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.401211 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.401108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xh7r\" (UniqueName: \"kubernetes.io/projected/000dab78-2620-41a2-a74c-e58809e8b312-kube-api-access-4xh7r\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.401211 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.401178    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.501852 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.501799    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502020 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.501870    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4xh7r\" (UniqueName: \"kubernetes.io/projected/000dab78-2620-41a2-a74c-e58809e8b312-kube-api-access-4xh7r\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502020 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.501902    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502020 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.501971    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502020 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.501990    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502190 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.502036    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000dab78-2620-41a2-a74c-e58809e8b312-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502274 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.502250    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502364 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.502340    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502407 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.502348    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.502446 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.502417    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.504559 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.504536    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000dab78-2620-41a2-a74c-e58809e8b312-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.509833 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.509801    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xh7r\" (UniqueName: \"kubernetes.io/projected/000dab78-2620-41a2-a74c-e58809e8b312-kube-api-access-4xh7r\") pod \"custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.609222 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.609116    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:52.756644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:52.756605    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"]
May 20 14:10:52.757491 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:10:52.757454    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000dab78_2620_41a2_a74c_e58809e8b312.slice/crio-189ecafb9f5e4bfe72db9cf85047e4873e1823fc60562ca3b522d7288942ddb6 WatchSource:0}: Error finding container 189ecafb9f5e4bfe72db9cf85047e4873e1823fc60562ca3b522d7288942ddb6: Status 404 returned error can't find the container with id 189ecafb9f5e4bfe72db9cf85047e4873e1823fc60562ca3b522d7288942ddb6
May 20 14:10:53.088239 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:53.088198    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerStarted","Data":"06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0"}
May 20 14:10:53.088239 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:53.088242    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerStarted","Data":"189ecafb9f5e4bfe72db9cf85047e4873e1823fc60562ca3b522d7288942ddb6"}
May 20 14:10:54.094194 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:54.094150    2554 generic.go:358] "Generic (PLEG): container finished" podID="000dab78-2620-41a2-a74c-e58809e8b312" containerID="06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0" exitCode=0
May 20 14:10:54.096214 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:54.094230    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerDied","Data":"06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0"}
May 20 14:10:55.101554 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:55.101517    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerStarted","Data":"d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9"}
May 20 14:10:55.101554 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:55.101557    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerStarted","Data":"9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d"}
May 20 14:10:55.101998 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:55.101693    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:10:55.123809 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:10:55.123745    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" podStartSLOduration=3.123727463 podStartE2EDuration="3.123727463s" podCreationTimestamp="2026-05-20 14:10:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:10:55.121518379 +0000 UTC m=+1239.744243447" watchObservedRunningTime="2026-05-20 14:10:55.123727463 +0000 UTC m=+1239.746452532"
May 20 14:11:02.610021 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:11:02.609981    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:11:02.610021 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:11:02.610024    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:11:02.612994 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:11:02.612966    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:11:03.137096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:11:03.137064    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:11:24.143270 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:11:24.143236    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:12:48.213014 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:48.212929    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"]
May 20 14:12:48.213525 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:48.213298    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="main" containerID="cri-o://9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d" gracePeriod=30
May 20 14:12:48.213525 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:48.213375    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="tokenizer" containerID="cri-o://d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9" gracePeriod=30
May 20 14:12:48.580159 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:48.580062    2554 generic.go:358] "Generic (PLEG): container finished" podID="000dab78-2620-41a2-a74c-e58809e8b312" containerID="9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d" exitCode=0
May 20 14:12:48.580159 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:48.580140    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerDied","Data":"9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d"}
May 20 14:12:49.585835 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.585807    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:12:49.587168 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.587147    2554 generic.go:358] "Generic (PLEG): container finished" podID="000dab78-2620-41a2-a74c-e58809e8b312" containerID="d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9" exitCode=0
May 20 14:12:49.587282 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.587194    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerDied","Data":"d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9"}
May 20 14:12:49.587282 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.587224    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx" event={"ID":"000dab78-2620-41a2-a74c-e58809e8b312","Type":"ContainerDied","Data":"189ecafb9f5e4bfe72db9cf85047e4873e1823fc60562ca3b522d7288942ddb6"}
May 20 14:12:49.587282 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.587243    2554 scope.go:117] "RemoveContainer" containerID="d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9"
May 20 14:12:49.599204 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.599099    2554 scope.go:117] "RemoveContainer" containerID="9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d"
May 20 14:12:49.608791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.608718    2554 scope.go:117] "RemoveContainer" containerID="06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0"
May 20 14:12:49.618549 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.618519    2554 scope.go:117] "RemoveContainer" containerID="d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9"
May 20 14:12:49.618876 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:12:49.618849    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9\": container with ID starting with d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9 not found: ID does not exist" containerID="d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9"
May 20 14:12:49.618965 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.618891    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9"} err="failed to get container status \"d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9\": rpc error: code = NotFound desc = could not find container \"d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9\": container with ID starting with d4e161a9372e61373b0b7f80d9c5085add068ccd33aa3cdfb66243e638ae46a9 not found: ID does not exist"
May 20 14:12:49.618965 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.618920    2554 scope.go:117] "RemoveContainer" containerID="9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d"
May 20 14:12:49.619193 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:12:49.619173    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d\": container with ID starting with 9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d not found: ID does not exist" containerID="9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d"
May 20 14:12:49.619261 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.619200    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d"} err="failed to get container status \"9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d\": rpc error: code = NotFound desc = could not find container \"9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d\": container with ID starting with 9c27f8ce72529be699782f6ba8e77fd46b76543bf003fd247f4d76a1793b970d not found: ID does not exist"
May 20 14:12:49.619261 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.619218    2554 scope.go:117] "RemoveContainer" containerID="06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0"
May 20 14:12:49.619465 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:12:49.619429    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0\": container with ID starting with 06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0 not found: ID does not exist" containerID="06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0"
May 20 14:12:49.619530 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.619468    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0"} err="failed to get container status \"06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0\": rpc error: code = NotFound desc = could not find container \"06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0\": container with ID starting with 06e6503cb255634c68c698793f7b8905a7d61007786bb29e75fa8d82a04631e0 not found: ID does not exist"
May 20 14:12:49.683936 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.683902    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-tmp\") pod \"000dab78-2620-41a2-a74c-e58809e8b312\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") "
May 20 14:12:49.684104 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.683977    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-uds\") pod \"000dab78-2620-41a2-a74c-e58809e8b312\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") "
May 20 14:12:49.684104 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684046    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-kserve-provision-location\") pod \"000dab78-2620-41a2-a74c-e58809e8b312\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") "
May 20 14:12:49.684104 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684092    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xh7r\" (UniqueName: \"kubernetes.io/projected/000dab78-2620-41a2-a74c-e58809e8b312-kube-api-access-4xh7r\") pod \"000dab78-2620-41a2-a74c-e58809e8b312\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") "
May 20 14:12:49.684293 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684128    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000dab78-2620-41a2-a74c-e58809e8b312-tls-certs\") pod \"000dab78-2620-41a2-a74c-e58809e8b312\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") "
May 20 14:12:49.684293 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684172    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-cache\") pod \"000dab78-2620-41a2-a74c-e58809e8b312\" (UID: \"000dab78-2620-41a2-a74c-e58809e8b312\") "
May 20 14:12:49.684409 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684288    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "000dab78-2620-41a2-a74c-e58809e8b312" (UID: "000dab78-2620-41a2-a74c-e58809e8b312"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:49.684409 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684300    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "000dab78-2620-41a2-a74c-e58809e8b312" (UID: "000dab78-2620-41a2-a74c-e58809e8b312"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:49.684513 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684481    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "000dab78-2620-41a2-a74c-e58809e8b312" (UID: "000dab78-2620-41a2-a74c-e58809e8b312"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:49.684573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684530    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:12:49.684573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684551    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:12:49.684888 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.684861    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "000dab78-2620-41a2-a74c-e58809e8b312" (UID: "000dab78-2620-41a2-a74c-e58809e8b312"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:49.686357 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.686332    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000dab78-2620-41a2-a74c-e58809e8b312-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "000dab78-2620-41a2-a74c-e58809e8b312" (UID: "000dab78-2620-41a2-a74c-e58809e8b312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:12:49.686357 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.686336    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000dab78-2620-41a2-a74c-e58809e8b312-kube-api-access-4xh7r" (OuterVolumeSpecName: "kube-api-access-4xh7r") pod "000dab78-2620-41a2-a74c-e58809e8b312" (UID: "000dab78-2620-41a2-a74c-e58809e8b312"). InnerVolumeSpecName "kube-api-access-4xh7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:12:49.785119 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.785074    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xh7r\" (UniqueName: \"kubernetes.io/projected/000dab78-2620-41a2-a74c-e58809e8b312-kube-api-access-4xh7r\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:12:49.785119 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.785114    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000dab78-2620-41a2-a74c-e58809e8b312-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:12:49.785119 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.785124    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:12:49.785365 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:49.785139    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/000dab78-2620-41a2-a74c-e58809e8b312-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:12:50.591863 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:50.591748    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"
May 20 14:12:50.618241 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:50.618200    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"]
May 20 14:12:50.627521 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:50.627490    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5c699bbd8m7bx"]
May 20 14:12:52.177295 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:12:52.177262    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000dab78-2620-41a2-a74c-e58809e8b312" path="/var/lib/kubelet/pods/000dab78-2620-41a2-a74c-e58809e8b312/volumes"
May 20 14:13:20.947324 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947286    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"]
May 20 14:13:20.947821 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947702    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="storage-initializer"
May 20 14:13:20.947821 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947716    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="storage-initializer"
May 20 14:13:20.947821 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947727    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="main"
May 20 14:13:20.947821 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947733    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="main"
May 20 14:13:20.947821 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947756    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="tokenizer"
May 20 14:13:20.947821 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947762    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="tokenizer"
May 20 14:13:20.947821 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947826    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="tokenizer"
May 20 14:13:20.948096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.947833    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="000dab78-2620-41a2-a74c-e58809e8b312" containerName="main"
May 20 14:13:20.952233 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.952212    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:20.954725 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.954701    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:13:20.954867 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.954701    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:13:20.954867 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.954711    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 20 14:13:20.955312 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.955294    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-l6btz\""
May 20 14:13:20.961223 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:20.961195    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"]
May 20 14:13:21.077987 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.077939    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.078208 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.078002    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.078208 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.078026    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.078208 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.078057    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7m58\" (UniqueName: \"kubernetes.io/projected/f5f0c55f-03af-4546-9141-b1b55786a6b3-kube-api-access-j7m58\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.078365 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.078195    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0c55f-03af-4546-9141-b1b55786a6b3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.078365 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.078239    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.178854 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.178798    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0c55f-03af-4546-9141-b1b55786a6b3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.178867    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.178907    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.178963    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.178986    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.179043    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j7m58\" (UniqueName: \"kubernetes.io/projected/f5f0c55f-03af-4546-9141-b1b55786a6b3-kube-api-access-j7m58\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179426 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.179382    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179550 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.179432    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179550 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.179480    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.179550 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.179529    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.181566 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.181547    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0c55f-03af-4546-9141-b1b55786a6b3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.194003 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.193971    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7m58\" (UniqueName: \"kubernetes.io/projected/f5f0c55f-03af-4546-9141-b1b55786a6b3-kube-api-access-j7m58\") pod \"router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.264077 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.263989    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:21.408683 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.408650    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"]
May 20 14:13:21.409480 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:13:21.409449    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f0c55f_03af_4546_9141_b1b55786a6b3.slice/crio-b2a43533ed588c27210f2d57b23aa3d0af740e0daf38457d4d5f010279aeb0b8 WatchSource:0}: Error finding container b2a43533ed588c27210f2d57b23aa3d0af740e0daf38457d4d5f010279aeb0b8: Status 404 returned error can't find the container with id b2a43533ed588c27210f2d57b23aa3d0af740e0daf38457d4d5f010279aeb0b8
May 20 14:13:21.411452 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.411427    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:13:21.727368 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.727330    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerStarted","Data":"25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f"}
May 20 14:13:21.727368 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:21.727370    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerStarted","Data":"b2a43533ed588c27210f2d57b23aa3d0af740e0daf38457d4d5f010279aeb0b8"}
May 20 14:13:22.733633 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:22.733596    2554 generic.go:358] "Generic (PLEG): container finished" podID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerID="25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f" exitCode=0
May 20 14:13:22.734067 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:22.733688    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerDied","Data":"25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f"}
May 20 14:13:23.739577 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:23.739533    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerStarted","Data":"f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12"}
May 20 14:13:23.739976 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:23.739584    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerStarted","Data":"5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2"}
May 20 14:13:23.739976 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:23.739608    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:23.770394 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:23.770338    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" podStartSLOduration=3.77032112 podStartE2EDuration="3.77032112s" podCreationTimestamp="2026-05-20 14:13:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:23.767914175 +0000 UTC m=+1388.390639249" watchObservedRunningTime="2026-05-20 14:13:23.77032112 +0000 UTC m=+1388.393046194"
May 20 14:13:31.265048 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:31.264999    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:31.265048 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:31.265057    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:31.267741 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:31.267717    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:31.780185 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:31.780152    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:13:52.784869 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:13:52.784806    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:15:07.413346 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:07.413304    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"]
May 20 14:15:07.413791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:07.413718    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="main" containerID="cri-o://5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2" gracePeriod=30
May 20 14:15:07.413791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:07.413748    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="tokenizer" containerID="cri-o://f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12" gracePeriod=30
May 20 14:15:08.187574 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.187539    2554 generic.go:358] "Generic (PLEG): container finished" podID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerID="5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2" exitCode=0
May 20 14:15:08.187740 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.187609    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerDied","Data":"5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2"}
May 20 14:15:08.774159 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.774133    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:15:08.785417 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785392    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-kserve-provision-location\") pod \"f5f0c55f-03af-4546-9141-b1b55786a6b3\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") "
May 20 14:15:08.785573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785424    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-tmp\") pod \"f5f0c55f-03af-4546-9141-b1b55786a6b3\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") "
May 20 14:15:08.785573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785447    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-uds\") pod \"f5f0c55f-03af-4546-9141-b1b55786a6b3\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") "
May 20 14:15:08.785573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785481    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7m58\" (UniqueName: \"kubernetes.io/projected/f5f0c55f-03af-4546-9141-b1b55786a6b3-kube-api-access-j7m58\") pod \"f5f0c55f-03af-4546-9141-b1b55786a6b3\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") "
May 20 14:15:08.785573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785497    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0c55f-03af-4546-9141-b1b55786a6b3-tls-certs\") pod \"f5f0c55f-03af-4546-9141-b1b55786a6b3\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") "
May 20 14:15:08.785573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785528    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-cache\") pod \"f5f0c55f-03af-4546-9141-b1b55786a6b3\" (UID: \"f5f0c55f-03af-4546-9141-b1b55786a6b3\") "
May 20 14:15:08.785868 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785800    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f5f0c55f-03af-4546-9141-b1b55786a6b3" (UID: "f5f0c55f-03af-4546-9141-b1b55786a6b3"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:08.785868 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785814    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f5f0c55f-03af-4546-9141-b1b55786a6b3" (UID: "f5f0c55f-03af-4546-9141-b1b55786a6b3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:08.785997 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.785919    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f5f0c55f-03af-4546-9141-b1b55786a6b3" (UID: "f5f0c55f-03af-4546-9141-b1b55786a6b3"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:08.786224 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.786206    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5f0c55f-03af-4546-9141-b1b55786a6b3" (UID: "f5f0c55f-03af-4546-9141-b1b55786a6b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:08.787603 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.787583    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f0c55f-03af-4546-9141-b1b55786a6b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5f0c55f-03af-4546-9141-b1b55786a6b3" (UID: "f5f0c55f-03af-4546-9141-b1b55786a6b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:15:08.787719 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.787699    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f0c55f-03af-4546-9141-b1b55786a6b3-kube-api-access-j7m58" (OuterVolumeSpecName: "kube-api-access-j7m58") pod "f5f0c55f-03af-4546-9141-b1b55786a6b3" (UID: "f5f0c55f-03af-4546-9141-b1b55786a6b3"). InnerVolumeSpecName "kube-api-access-j7m58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:15:08.886660 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.886567    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7m58\" (UniqueName: \"kubernetes.io/projected/f5f0c55f-03af-4546-9141-b1b55786a6b3-kube-api-access-j7m58\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:15:08.886660 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.886601    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f0c55f-03af-4546-9141-b1b55786a6b3-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:15:08.886660 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.886617    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:15:08.886660 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.886630    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:15:08.886660 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.886644    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:15:08.886660 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:08.886657    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f5f0c55f-03af-4546-9141-b1b55786a6b3-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:15:09.193564 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.193532    2554 generic.go:358] "Generic (PLEG): container finished" podID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerID="f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12" exitCode=0
May 20 14:15:09.193769 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.193590    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerDied","Data":"f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12"}
May 20 14:15:09.193769 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.193611    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"
May 20 14:15:09.193769 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.193639    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc" event={"ID":"f5f0c55f-03af-4546-9141-b1b55786a6b3","Type":"ContainerDied","Data":"b2a43533ed588c27210f2d57b23aa3d0af740e0daf38457d4d5f010279aeb0b8"}
May 20 14:15:09.193769 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.193658    2554 scope.go:117] "RemoveContainer" containerID="f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12"
May 20 14:15:09.204423 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.204402    2554 scope.go:117] "RemoveContainer" containerID="5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2"
May 20 14:15:09.212996 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.212974    2554 scope.go:117] "RemoveContainer" containerID="25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f"
May 20 14:15:09.218429 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.218399    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"]
May 20 14:15:09.221263 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.221241    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-6dbd7f8f9b-tdmjc"]
May 20 14:15:09.223247 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.223223    2554 scope.go:117] "RemoveContainer" containerID="f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12"
May 20 14:15:09.223544 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:15:09.223517    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12\": container with ID starting with f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12 not found: ID does not exist" containerID="f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12"
May 20 14:15:09.223617 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.223547    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12"} err="failed to get container status \"f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12\": rpc error: code = NotFound desc = could not find container \"f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12\": container with ID starting with f7229cdcaad724dbdc8242b776171d6581f291fe0c6d5a198e2244a66e929f12 not found: ID does not exist"
May 20 14:15:09.223617 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.223567    2554 scope.go:117] "RemoveContainer" containerID="5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2"
May 20 14:15:09.223833 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:15:09.223814    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2\": container with ID starting with 5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2 not found: ID does not exist" containerID="5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2"
May 20 14:15:09.223887 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.223865    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2"} err="failed to get container status \"5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2\": rpc error: code = NotFound desc = could not find container \"5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2\": container with ID starting with 5d65eaa5447c9634e104d3875126c78565eb7cf72fb9452f32e4f84550d02be2 not found: ID does not exist"
May 20 14:15:09.223922 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.223889    2554 scope.go:117] "RemoveContainer" containerID="25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f"
May 20 14:15:09.224123 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:15:09.224106    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f\": container with ID starting with 25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f not found: ID does not exist" containerID="25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f"
May 20 14:15:09.224179 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:09.224129    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f"} err="failed to get container status \"25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f\": rpc error: code = NotFound desc = could not find container \"25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f\": container with ID starting with 25fc2453f634e8232c694158bd12f2008359b5128ff5a55ce225b50878cb560f not found: ID does not exist"
May 20 14:15:10.176976 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:10.176938    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" path="/var/lib/kubelet/pods/f5f0c55f-03af-4546-9141-b1b55786a6b3/volumes"
May 20 14:15:16.014828 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.014794    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:15:16.018756 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.018731    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:15:16.578706 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.578669    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"]
May 20 14:15:16.579136 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579120    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="tokenizer"
May 20 14:15:16.579199 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579138    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="tokenizer"
May 20 14:15:16.579199 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579150    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="storage-initializer"
May 20 14:15:16.579199 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579157    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="storage-initializer"
May 20 14:15:16.579199 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579169    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="main"
May 20 14:15:16.579199 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579175    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="main"
May 20 14:15:16.579378 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579250    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="main"
May 20 14:15:16.579378 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.579260    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5f0c55f-03af-4546-9141-b1b55786a6b3" containerName="tokenizer"
May 20 14:15:16.583750 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.583728    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.586282 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.586259    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:15:16.586385 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.586316    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:15:16.586940 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.586920    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-6dddh\""
May 20 14:15:16.587043 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.586962    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 20 14:15:16.595827 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.595800    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"]
May 20 14:15:16.660331 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.660300    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.660331 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.660335    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.660802 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.660429    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwwc\" (UniqueName: \"kubernetes.io/projected/a74bf79e-2c29-46c0-b6a1-f769085ee575-kube-api-access-zcwwc\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.660802 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.660488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74bf79e-2c29-46c0-b6a1-f769085ee575-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.660802 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.660515    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.660802 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.660536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761059 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761018    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwwc\" (UniqueName: \"kubernetes.io/projected/a74bf79e-2c29-46c0-b6a1-f769085ee575-kube-api-access-zcwwc\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761264 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761072    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74bf79e-2c29-46c0-b6a1-f769085ee575-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761362 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761254    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761362 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761312    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761467 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761396    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761467 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761441    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761722 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761695    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761776 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761737    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761776 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761764    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.761877 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.761783    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.763574 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.763549    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74bf79e-2c29-46c0-b6a1-f769085ee575-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.768912 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.768886    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwwc\" (UniqueName: \"kubernetes.io/projected/a74bf79e-2c29-46c0-b6a1-f769085ee575-kube-api-access-zcwwc\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:16.895289 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:16.895190    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:17.062870 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:17.062820    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"]
May 20 14:15:17.063573 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:15:17.063544    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74bf79e_2c29_46c0_b6a1_f769085ee575.slice/crio-09301a53dbcdd1afccb6345bff706508fce9ed454da8d22f73fcf21d2aec9915 WatchSource:0}: Error finding container 09301a53dbcdd1afccb6345bff706508fce9ed454da8d22f73fcf21d2aec9915: Status 404 returned error can't find the container with id 09301a53dbcdd1afccb6345bff706508fce9ed454da8d22f73fcf21d2aec9915
May 20 14:15:17.231307 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:17.231266    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerStarted","Data":"f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b"}
May 20 14:15:17.231307 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:17.231306    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerStarted","Data":"09301a53dbcdd1afccb6345bff706508fce9ed454da8d22f73fcf21d2aec9915"}
May 20 14:15:18.236745 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:18.236653    2554 generic.go:358] "Generic (PLEG): container finished" podID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerID="f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b" exitCode=0
May 20 14:15:18.237133 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:18.236740    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerDied","Data":"f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b"}
May 20 14:15:19.243105 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:19.243061    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerStarted","Data":"55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334"}
May 20 14:15:19.243105 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:19.243105    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerStarted","Data":"9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8"}
May 20 14:15:19.243630 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:19.243183    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:19.269605 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:19.269548    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" podStartSLOduration=3.26953128 podStartE2EDuration="3.26953128s" podCreationTimestamp="2026-05-20 14:15:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:15:19.26569347 +0000 UTC m=+1503.888418541" watchObservedRunningTime="2026-05-20 14:15:19.26953128 +0000 UTC m=+1503.892256345"
May 20 14:15:26.896352 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:26.896302    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:26.896352 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:26.896360    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:26.899347 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:26.899317    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:27.287189 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:27.287159    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:15:48.291346 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:15:48.291260    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:17:55.383611 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:55.383572    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"]
May 20 14:17:55.384164 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:55.383901    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="main" containerID="cri-o://9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8" gracePeriod=30
May 20 14:17:55.384164 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:55.383946    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="tokenizer" containerID="cri-o://55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334" gracePeriod=30
May 20 14:17:55.912685 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:55.912643    2554 generic.go:358] "Generic (PLEG): container finished" podID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerID="9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8" exitCode=0
May 20 14:17:55.912877 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:55.912718    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerDied","Data":"9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8"}
May 20 14:17:56.749442 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.749412    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:17:56.861021 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.860928    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcwwc\" (UniqueName: \"kubernetes.io/projected/a74bf79e-2c29-46c0-b6a1-f769085ee575-kube-api-access-zcwwc\") pod \"a74bf79e-2c29-46c0-b6a1-f769085ee575\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") "
May 20 14:17:56.861205 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861023    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-uds\") pod \"a74bf79e-2c29-46c0-b6a1-f769085ee575\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") "
May 20 14:17:56.861205 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861084    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74bf79e-2c29-46c0-b6a1-f769085ee575-tls-certs\") pod \"a74bf79e-2c29-46c0-b6a1-f769085ee575\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") "
May 20 14:17:56.861205 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861115    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-kserve-provision-location\") pod \"a74bf79e-2c29-46c0-b6a1-f769085ee575\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") "
May 20 14:17:56.861205 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861166    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-cache\") pod \"a74bf79e-2c29-46c0-b6a1-f769085ee575\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") "
May 20 14:17:56.861405 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861230    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-tmp\") pod \"a74bf79e-2c29-46c0-b6a1-f769085ee575\" (UID: \"a74bf79e-2c29-46c0-b6a1-f769085ee575\") "
May 20 14:17:56.861405 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861265    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a74bf79e-2c29-46c0-b6a1-f769085ee575" (UID: "a74bf79e-2c29-46c0-b6a1-f769085ee575"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:17:56.861496 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861426    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a74bf79e-2c29-46c0-b6a1-f769085ee575" (UID: "a74bf79e-2c29-46c0-b6a1-f769085ee575"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:17:56.861594 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861576    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:17:56.861636 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861600    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:17:56.861672 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861632    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "a74bf79e-2c29-46c0-b6a1-f769085ee575" (UID: "a74bf79e-2c29-46c0-b6a1-f769085ee575"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:17:56.861965 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.861940    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a74bf79e-2c29-46c0-b6a1-f769085ee575" (UID: "a74bf79e-2c29-46c0-b6a1-f769085ee575"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:17:56.863285 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.863266    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74bf79e-2c29-46c0-b6a1-f769085ee575-kube-api-access-zcwwc" (OuterVolumeSpecName: "kube-api-access-zcwwc") pod "a74bf79e-2c29-46c0-b6a1-f769085ee575" (UID: "a74bf79e-2c29-46c0-b6a1-f769085ee575"). InnerVolumeSpecName "kube-api-access-zcwwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:17:56.863285 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.863271    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74bf79e-2c29-46c0-b6a1-f769085ee575-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a74bf79e-2c29-46c0-b6a1-f769085ee575" (UID: "a74bf79e-2c29-46c0-b6a1-f769085ee575"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:17:56.921498 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.921464    2554 generic.go:358] "Generic (PLEG): container finished" podID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerID="55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334" exitCode=0
May 20 14:17:56.921662 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.921551    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"
May 20 14:17:56.921662 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.921553    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerDied","Data":"55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334"}
May 20 14:17:56.921743 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.921662    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898" event={"ID":"a74bf79e-2c29-46c0-b6a1-f769085ee575","Type":"ContainerDied","Data":"09301a53dbcdd1afccb6345bff706508fce9ed454da8d22f73fcf21d2aec9915"}
May 20 14:17:56.921743 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.921691    2554 scope.go:117] "RemoveContainer" containerID="55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334"
May 20 14:17:56.931176 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.931155    2554 scope.go:117] "RemoveContainer" containerID="9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8"
May 20 14:17:56.939447 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.939431    2554 scope.go:117] "RemoveContainer" containerID="f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b"
May 20 14:17:56.944587 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.944562    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"]
May 20 14:17:56.949604 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.949579    2554 scope.go:117] "RemoveContainer" containerID="55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334"
May 20 14:17:56.949938 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:17:56.949915    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334\": container with ID starting with 55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334 not found: ID does not exist" containerID="55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334"
May 20 14:17:56.950019 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.949952    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334"} err="failed to get container status \"55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334\": rpc error: code = NotFound desc = could not find container \"55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334\": container with ID starting with 55d380f4d97bea59a2e9b5561be4521d5eed329d5891de4f0ca778f520bb5334 not found: ID does not exist"
May 20 14:17:56.950019 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.949981    2554 scope.go:117] "RemoveContainer" containerID="9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8"
May 20 14:17:56.950262 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:17:56.950245    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8\": container with ID starting with 9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8 not found: ID does not exist" containerID="9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8"
May 20 14:17:56.950324 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.950270    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8"} err="failed to get container status \"9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8\": rpc error: code = NotFound desc = could not find container \"9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8\": container with ID starting with 9bcace174b78b0e10c0789f106eac707b7b01df558337183f825031e4ffb8bb8 not found: ID does not exist"
May 20 14:17:56.950324 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.950289    2554 scope.go:117] "RemoveContainer" containerID="f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b"
May 20 14:17:56.950531 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:17:56.950513    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b\": container with ID starting with f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b not found: ID does not exist" containerID="f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b"
May 20 14:17:56.950582 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.950541    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b"} err="failed to get container status \"f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b\": rpc error: code = NotFound desc = could not find container \"f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b\": container with ID starting with f2505bf5bd9535772e92fff914b73bd3ca03b23356b5079adb03aaf4bb539c9b not found: ID does not exist"
May 20 14:17:56.951554 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.951533    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-schedv898"]
May 20 14:17:56.962637 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.962599    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwwc\" (UniqueName: \"kubernetes.io/projected/a74bf79e-2c29-46c0-b6a1-f769085ee575-kube-api-access-zcwwc\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:17:56.962637 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.962632    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74bf79e-2c29-46c0-b6a1-f769085ee575-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:17:56.962637 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.962643    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:17:56.962867 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:56.962653    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a74bf79e-2c29-46c0-b6a1-f769085ee575-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:17:58.177135 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:17:58.177100    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" path="/var/lib/kubelet/pods/a74bf79e-2c29-46c0-b6a1-f769085ee575/volumes"
May 20 14:18:03.043352 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.043313    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"]
May 20 14:18:03.043984 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.043960    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="tokenizer"
May 20 14:18:03.044064 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.043987    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="tokenizer"
May 20 14:18:03.044064 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.044031    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="main"
May 20 14:18:03.044064 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.044042    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="main"
May 20 14:18:03.044064 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.044058    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="storage-initializer"
May 20 14:18:03.044288 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.044067    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="storage-initializer"
May 20 14:18:03.044288 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.044173    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="tokenizer"
May 20 14:18:03.044288 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.044191    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="a74bf79e-2c29-46c0-b6a1-f769085ee575" containerName="main"
May 20 14:18:03.050037 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.050008    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.052578 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.052552    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:18:03.052727 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.052562    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 20 14:18:03.053259 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.053237    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:18:03.053367 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.053237    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-dbl2j\""
May 20 14:18:03.059434 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.059409    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"]
May 20 14:18:03.217008 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.216966    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a062114-9ead-4fbf-bf52-24437f148e05-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.217226 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.217017    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.217226 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.217213    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.217382 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.217245    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.217382 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.217276    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxgh\" (UniqueName: \"kubernetes.io/projected/4a062114-9ead-4fbf-bf52-24437f148e05-kube-api-access-pvxgh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.217382 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.217316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.318146 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.318099    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.318364 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.318234    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.318364 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.318278    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxgh\" (UniqueName: \"kubernetes.io/projected/4a062114-9ead-4fbf-bf52-24437f148e05-kube-api-access-pvxgh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.318364 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.318325    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.318544 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.318376    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a062114-9ead-4fbf-bf52-24437f148e05-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.318544 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.318470    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.318544 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.318489    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.319142 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.319107    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.319142 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.319135    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-cache\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.319344 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.319170    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-tmp\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.327232 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.327204    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a062114-9ead-4fbf-bf52-24437f148e05-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.329253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.329229    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxgh\" (UniqueName: \"kubernetes.io/projected/4a062114-9ead-4fbf-bf52-24437f148e05-kube-api-access-pvxgh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.362555 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.362515    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:03.510783 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.510747    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"]
May 20 14:18:03.513744 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:18:03.513718    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a062114_9ead_4fbf_bf52_24437f148e05.slice/crio-e0003cca6b9ed4df3219ce434940b57002be5211a276906eedab14c7984798db WatchSource:0}: Error finding container e0003cca6b9ed4df3219ce434940b57002be5211a276906eedab14c7984798db: Status 404 returned error can't find the container with id e0003cca6b9ed4df3219ce434940b57002be5211a276906eedab14c7984798db
May 20 14:18:03.952478 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.952426    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerStarted","Data":"fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd"}
May 20 14:18:03.952478 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:03.952484    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerStarted","Data":"e0003cca6b9ed4df3219ce434940b57002be5211a276906eedab14c7984798db"}
May 20 14:18:04.960523 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:04.960486    2554 generic.go:358] "Generic (PLEG): container finished" podID="4a062114-9ead-4fbf-bf52-24437f148e05" containerID="fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd" exitCode=0
May 20 14:18:04.960924 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:04.960576    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerDied","Data":"fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd"}
May 20 14:18:05.967642 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:05.967607    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerStarted","Data":"2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421"}
May 20 14:18:05.967642 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:05.967647    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerStarted","Data":"2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87"}
May 20 14:18:05.968079 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:05.967819    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:05.989511 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:05.989455    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" podStartSLOduration=2.989440812 podStartE2EDuration="2.989440812s" podCreationTimestamp="2026-05-20 14:18:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:18:05.986620478 +0000 UTC m=+1670.609345570" watchObservedRunningTime="2026-05-20 14:18:05.989440812 +0000 UTC m=+1670.612165880"
May 20 14:18:13.363096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:13.363045    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:13.363547 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:13.363110    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:13.365958 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:13.365934    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:14.011745 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:14.011715    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:18:35.016731 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:18:35.016694    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:20:16.057105 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:20:16.057071    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:20:16.064805 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:20:16.064764    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:21:51.099710 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.099617    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"]
May 20 14:21:51.100307 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.100112    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="main" containerID="cri-o://2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87" gracePeriod=30
May 20 14:21:51.100307 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.100158    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="tokenizer" containerID="cri-o://2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421" gracePeriod=30
May 20 14:21:51.199054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.199012    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"]
May 20 14:21:51.203650 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.203610    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.206605 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.206577    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-28b2d\""
May 20 14:21:51.214872 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.214819    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"]
May 20 14:21:51.334306 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvbm\" (UniqueName: \"kubernetes.io/projected/60bf9e3c-3d11-4364-817a-33f00307fc7b-kube-api-access-jfvbm\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334501 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334320    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf9e3c-3d11-4364-817a-33f00307fc7b-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334501 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334400    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334501 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334433    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334501 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334478    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334501 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334504    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334704 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334595    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334704 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334651    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.334704 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.334697    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436256 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436208    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436280    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436318    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436357    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436391    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvbm\" (UniqueName: \"kubernetes.io/projected/60bf9e3c-3d11-4364-817a-33f00307fc7b-kube-api-access-jfvbm\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436451    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf9e3c-3d11-4364-817a-33f00307fc7b-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436790 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436496    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436790 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436546    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436790 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436591    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.436790 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.436743    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.439371 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.437314    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.439371 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.437401    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bf9e3c-3d11-4364-817a-33f00307fc7b-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.439371 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.437640    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.439371 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.437752    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.442725 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.442699    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.443429 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.443402    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.445240 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.445214    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvbm\" (UniqueName: \"kubernetes.io/projected/60bf9e3c-3d11-4364-817a-33f00307fc7b-kube-api-access-jfvbm\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.445426 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.445410    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bf9e3c-3d11-4364-817a-33f00307fc7b-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-x7rzn\" (UID: \"60bf9e3c-3d11-4364-817a-33f00307fc7b\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.522135 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.522090    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:51.694073 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:21:51.694035    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bf9e3c_3d11_4364_817a_33f00307fc7b.slice/crio-fff3ad768a1f3c68aa53d79101c603fd61610ee3b8ea20e50db0c6bebd07ac4f WatchSource:0}: Error finding container fff3ad768a1f3c68aa53d79101c603fd61610ee3b8ea20e50db0c6bebd07ac4f: Status 404 returned error can't find the container with id fff3ad768a1f3c68aa53d79101c603fd61610ee3b8ea20e50db0c6bebd07ac4f
May 20 14:21:51.695659 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.695524    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"]
May 20 14:21:51.696649 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.696623    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:21:51.697191 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.697088    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:21:51.697284 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.697194    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:21:51.697284 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.697241    2554 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:21:51.942768 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.942671    2554 generic.go:358] "Generic (PLEG): container finished" podID="4a062114-9ead-4fbf-bf52-24437f148e05" containerID="2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87" exitCode=0
May 20 14:21:51.942974 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.942886    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerDied","Data":"2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87"}
May 20 14:21:51.944602 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.944562    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn" event={"ID":"60bf9e3c-3d11-4364-817a-33f00307fc7b","Type":"ContainerStarted","Data":"777c89f74890ed4a547a0ceae5cbfae815c1173f16b67ca695ddc02ae3a38e99"}
May 20 14:21:51.944602 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.944599    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn" event={"ID":"60bf9e3c-3d11-4364-817a-33f00307fc7b","Type":"ContainerStarted","Data":"fff3ad768a1f3c68aa53d79101c603fd61610ee3b8ea20e50db0c6bebd07ac4f"}
May 20 14:21:51.968024 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:51.967969    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn" podStartSLOduration=0.967952429 podStartE2EDuration="967.952429ms" podCreationTimestamp="2026-05-20 14:21:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:21:51.964567873 +0000 UTC m=+1896.587292952" watchObservedRunningTime="2026-05-20 14:21:51.967952429 +0000 UTC m=+1896.590677559"
May 20 14:21:52.523013 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.522969    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:52.528735 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.528702    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:52.569426 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.569398    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:21:52.648746 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.648642    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a062114-9ead-4fbf-bf52-24437f148e05-tls-certs\") pod \"4a062114-9ead-4fbf-bf52-24437f148e05\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") "
May 20 14:21:52.648746 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.648728    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-cache\") pod \"4a062114-9ead-4fbf-bf52-24437f148e05\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") "
May 20 14:21:52.649019 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.648764    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-kserve-provision-location\") pod \"4a062114-9ead-4fbf-bf52-24437f148e05\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") "
May 20 14:21:52.649019 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.648796    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-tmp\") pod \"4a062114-9ead-4fbf-bf52-24437f148e05\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") "
May 20 14:21:52.649019 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.648831    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-uds\") pod \"4a062114-9ead-4fbf-bf52-24437f148e05\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") "
May 20 14:21:52.649019 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.648870    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvxgh\" (UniqueName: \"kubernetes.io/projected/4a062114-9ead-4fbf-bf52-24437f148e05-kube-api-access-pvxgh\") pod \"4a062114-9ead-4fbf-bf52-24437f148e05\" (UID: \"4a062114-9ead-4fbf-bf52-24437f148e05\") "
May 20 14:21:52.649249 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.649074    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "4a062114-9ead-4fbf-bf52-24437f148e05" (UID: "4a062114-9ead-4fbf-bf52-24437f148e05"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:21:52.649249 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.649203    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4a062114-9ead-4fbf-bf52-24437f148e05" (UID: "4a062114-9ead-4fbf-bf52-24437f148e05"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:21:52.649365 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.649273    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:21:52.649365 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.649292    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:21:52.649473 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.649366    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "4a062114-9ead-4fbf-bf52-24437f148e05" (UID: "4a062114-9ead-4fbf-bf52-24437f148e05"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:21:52.649559 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.649530    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a062114-9ead-4fbf-bf52-24437f148e05" (UID: "4a062114-9ead-4fbf-bf52-24437f148e05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:21:52.651166 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.651143    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a062114-9ead-4fbf-bf52-24437f148e05-kube-api-access-pvxgh" (OuterVolumeSpecName: "kube-api-access-pvxgh") pod "4a062114-9ead-4fbf-bf52-24437f148e05" (UID: "4a062114-9ead-4fbf-bf52-24437f148e05"). InnerVolumeSpecName "kube-api-access-pvxgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:21:52.651537 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.651516    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a062114-9ead-4fbf-bf52-24437f148e05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a062114-9ead-4fbf-bf52-24437f148e05" (UID: "4a062114-9ead-4fbf-bf52-24437f148e05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:21:52.750047 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.750001    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvxgh\" (UniqueName: \"kubernetes.io/projected/4a062114-9ead-4fbf-bf52-24437f148e05-kube-api-access-pvxgh\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:21:52.750047 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.750041    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a062114-9ead-4fbf-bf52-24437f148e05-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:21:52.750256 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.750057    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:21:52.750256 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.750071    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a062114-9ead-4fbf-bf52-24437f148e05-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:21:52.950975 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.950944    2554 generic.go:358] "Generic (PLEG): container finished" podID="4a062114-9ead-4fbf-bf52-24437f148e05" containerID="2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421" exitCode=0
May 20 14:21:52.951168 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.951025    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"
May 20 14:21:52.951168 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.951026    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerDied","Data":"2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421"}
May 20 14:21:52.951168 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.951072    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2" event={"ID":"4a062114-9ead-4fbf-bf52-24437f148e05","Type":"ContainerDied","Data":"e0003cca6b9ed4df3219ce434940b57002be5211a276906eedab14c7984798db"}
May 20 14:21:52.951168 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.951095    2554 scope.go:117] "RemoveContainer" containerID="2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421"
May 20 14:21:52.951987 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.951957    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:52.953183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.953157    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-x7rzn"
May 20 14:21:52.964159 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.964093    2554 scope.go:117] "RemoveContainer" containerID="2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87"
May 20 14:21:52.975092 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.975067    2554 scope.go:117] "RemoveContainer" containerID="fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd"
May 20 14:21:52.989362 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.989333    2554 scope.go:117] "RemoveContainer" containerID="2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421"
May 20 14:21:52.989827 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:21:52.989782    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421\": container with ID starting with 2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421 not found: ID does not exist" containerID="2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421"
May 20 14:21:52.990352 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.990292    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421"} err="failed to get container status \"2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421\": rpc error: code = NotFound desc = could not find container \"2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421\": container with ID starting with 2745a438c0df86681cf44b62475167f07146ce6d6865b643d823154b8c00c421 not found: ID does not exist"
May 20 14:21:52.990352 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.990337    2554 scope.go:117] "RemoveContainer" containerID="2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87"
May 20 14:21:52.990716 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:21:52.990686    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87\": container with ID starting with 2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87 not found: ID does not exist" containerID="2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87"
May 20 14:21:52.990894 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.990723    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87"} err="failed to get container status \"2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87\": rpc error: code = NotFound desc = could not find container \"2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87\": container with ID starting with 2d5c8520b92b284a8f502fc3717b0b6eadfc222889629672391768cc1cee3a87 not found: ID does not exist"
May 20 14:21:52.990894 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.990748    2554 scope.go:117] "RemoveContainer" containerID="fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd"
May 20 14:21:52.991090 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:21:52.991067    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd\": container with ID starting with fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd not found: ID does not exist" containerID="fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd"
May 20 14:21:52.991170 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:52.991097    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd"} err="failed to get container status \"fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd\": rpc error: code = NotFound desc = could not find container \"fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd\": container with ID starting with fbbd2a4402b2e6ece14c817ffeb429062de06145cb72635aa427835f9c4fbcfd not found: ID does not exist"
May 20 14:21:53.005791 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:53.005750    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"]
May 20 14:21:53.006050 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:53.005810    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-5cb7f48br2"]
May 20 14:21:54.177374 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:21:54.177340    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" path="/var/lib/kubelet/pods/4a062114-9ead-4fbf-bf52-24437f148e05/volumes"
May 20 14:22:05.142079 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142046    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"]
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142472    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="storage-initializer"
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142483    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="storage-initializer"
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142492    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="main"
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142498    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="main"
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142508    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="tokenizer"
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142514    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="tokenizer"
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142588    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="main"
May 20 14:22:05.144414 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.142597    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="4a062114-9ead-4fbf-bf52-24437f148e05" containerName="tokenizer"
May 20 14:22:05.145742 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.145725    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.148142 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.148116    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:22:05.148270 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.148184    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-rnqnk\""
May 20 14:22:05.148855 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.148824    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 20 14:22:05.148951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.148875    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:22:05.156709 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.156682    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"]
May 20 14:22:05.167448 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.167418    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.167588 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.167476    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.167588 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.167513    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.167588 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.167543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70433e3d-b641-45e6-aa58-1d2a185b098a-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.167588 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.167569    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4j6c\" (UniqueName: \"kubernetes.io/projected/70433e3d-b641-45e6-aa58-1d2a185b098a-kube-api-access-s4j6c\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.167749 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.167603    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.270886 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.270822    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.270921    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.270969    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.271010    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70433e3d-b641-45e6-aa58-1d2a185b098a-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271096 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.271039    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4j6c\" (UniqueName: \"kubernetes.io/projected/70433e3d-b641-45e6-aa58-1d2a185b098a-kube-api-access-s4j6c\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271336 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.271307    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-tmp\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271399 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.271320    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271399 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.271350    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271490 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.271352    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-cache\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.271543 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.271523    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.273673 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.273649    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70433e3d-b641-45e6-aa58-1d2a185b098a-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.279322 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.279280    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4j6c\" (UniqueName: \"kubernetes.io/projected/70433e3d-b641-45e6-aa58-1d2a185b098a-kube-api-access-s4j6c\") pod \"router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.456416 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.456373    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:05.613316 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:05.613279    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"]
May 20 14:22:05.614814 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:22:05.614774    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70433e3d_b641_45e6_aa58_1d2a185b098a.slice/crio-17906d1f8fe09bafb3d2cbf81242657b5e52ec53c5eeee3be5f1485f49fa6381 WatchSource:0}: Error finding container 17906d1f8fe09bafb3d2cbf81242657b5e52ec53c5eeee3be5f1485f49fa6381: Status 404 returned error can't find the container with id 17906d1f8fe09bafb3d2cbf81242657b5e52ec53c5eeee3be5f1485f49fa6381
May 20 14:22:06.007312 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:06.007268    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerStarted","Data":"4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677"}
May 20 14:22:06.007312 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:06.007313    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerStarted","Data":"17906d1f8fe09bafb3d2cbf81242657b5e52ec53c5eeee3be5f1485f49fa6381"}
May 20 14:22:07.012653 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:07.012615    2554 generic.go:358] "Generic (PLEG): container finished" podID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerID="4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677" exitCode=0
May 20 14:22:07.013082 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:07.012677    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerDied","Data":"4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677"}
May 20 14:22:08.020027 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:08.019989    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerStarted","Data":"e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673"}
May 20 14:22:08.020027 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:08.020031    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerStarted","Data":"a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96"}
May 20 14:22:08.020474 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:08.020121    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:08.046401 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:08.046334    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" podStartSLOduration=3.046317782 podStartE2EDuration="3.046317782s" podCreationTimestamp="2026-05-20 14:22:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:22:08.043266383 +0000 UTC m=+1912.665991473" watchObservedRunningTime="2026-05-20 14:22:08.046317782 +0000 UTC m=+1912.669042850"
May 20 14:22:15.457282 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:15.457240    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:15.457282 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:15.457289    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:15.459982 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:15.459957    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:16.056926 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:16.056898    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:22:37.061778 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:22:37.061746    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:24:47.194069 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:47.193983    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"]
May 20 14:24:47.194721 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:47.194365    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="main" containerID="cri-o://a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96" gracePeriod=30
May 20 14:24:47.194721 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:47.194543    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="tokenizer" containerID="cri-o://e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673" gracePeriod=30
May 20 14:24:47.697640 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:47.697606    2554 generic.go:358] "Generic (PLEG): container finished" podID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerID="a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96" exitCode=0
May 20 14:24:47.697806 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:47.697684    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerDied","Data":"a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96"}
May 20 14:24:48.566989 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.566963    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:24:48.704143 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.704107    2554 generic.go:358] "Generic (PLEG): container finished" podID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerID="e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673" exitCode=0
May 20 14:24:48.704323 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.704170    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerDied","Data":"e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673"}
May 20 14:24:48.704323 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.704196    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm" event={"ID":"70433e3d-b641-45e6-aa58-1d2a185b098a","Type":"ContainerDied","Data":"17906d1f8fe09bafb3d2cbf81242657b5e52ec53c5eeee3be5f1485f49fa6381"}
May 20 14:24:48.704323 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.704201    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"
May 20 14:24:48.704323 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.704214    2554 scope.go:117] "RemoveContainer" containerID="e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673"
May 20 14:24:48.713581 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.713560    2554 scope.go:117] "RemoveContainer" containerID="a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96"
May 20 14:24:48.722089 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.722066    2554 scope.go:117] "RemoveContainer" containerID="4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677"
May 20 14:24:48.722786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.722748    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-tmp\") pod \"70433e3d-b641-45e6-aa58-1d2a185b098a\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") "
May 20 14:24:48.722949 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.722818    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-cache\") pod \"70433e3d-b641-45e6-aa58-1d2a185b098a\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") "
May 20 14:24:48.722949 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.722873    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70433e3d-b641-45e6-aa58-1d2a185b098a-tls-certs\") pod \"70433e3d-b641-45e6-aa58-1d2a185b098a\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") "
May 20 14:24:48.722949 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.722898    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4j6c\" (UniqueName: \"kubernetes.io/projected/70433e3d-b641-45e6-aa58-1d2a185b098a-kube-api-access-s4j6c\") pod \"70433e3d-b641-45e6-aa58-1d2a185b098a\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") "
May 20 14:24:48.723183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.722973    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-uds\") pod \"70433e3d-b641-45e6-aa58-1d2a185b098a\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") "
May 20 14:24:48.723183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723020    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-kserve-provision-location\") pod \"70433e3d-b641-45e6-aa58-1d2a185b098a\" (UID: \"70433e3d-b641-45e6-aa58-1d2a185b098a\") "
May 20 14:24:48.723183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723100    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "70433e3d-b641-45e6-aa58-1d2a185b098a" (UID: "70433e3d-b641-45e6-aa58-1d2a185b098a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:24:48.723183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723107    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "70433e3d-b641-45e6-aa58-1d2a185b098a" (UID: "70433e3d-b641-45e6-aa58-1d2a185b098a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:24:48.723368 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723296    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "70433e3d-b641-45e6-aa58-1d2a185b098a" (UID: "70433e3d-b641-45e6-aa58-1d2a185b098a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:24:48.723438 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723416    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:24:48.723493 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723437    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:24:48.723493 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723453    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:24:48.723878 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.723818    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70433e3d-b641-45e6-aa58-1d2a185b098a" (UID: "70433e3d-b641-45e6-aa58-1d2a185b098a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:24:48.725183 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.725158    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70433e3d-b641-45e6-aa58-1d2a185b098a-kube-api-access-s4j6c" (OuterVolumeSpecName: "kube-api-access-s4j6c") pod "70433e3d-b641-45e6-aa58-1d2a185b098a" (UID: "70433e3d-b641-45e6-aa58-1d2a185b098a"). InnerVolumeSpecName "kube-api-access-s4j6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:24:48.725278 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.725161    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70433e3d-b641-45e6-aa58-1d2a185b098a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70433e3d-b641-45e6-aa58-1d2a185b098a" (UID: "70433e3d-b641-45e6-aa58-1d2a185b098a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:24:48.744669 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.744642    2554 scope.go:117] "RemoveContainer" containerID="e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673"
May 20 14:24:48.745014 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:24:48.744986    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673\": container with ID starting with e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673 not found: ID does not exist" containerID="e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673"
May 20 14:24:48.745146 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.745023    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673"} err="failed to get container status \"e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673\": rpc error: code = NotFound desc = could not find container \"e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673\": container with ID starting with e6730af8cfcb399363e0cd13430ea6cda60aefc9a7e2793ef1ae905846b25673 not found: ID does not exist"
May 20 14:24:48.745146 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.745044    2554 scope.go:117] "RemoveContainer" containerID="a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96"
May 20 14:24:48.745348 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:24:48.745332    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96\": container with ID starting with a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96 not found: ID does not exist" containerID="a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96"
May 20 14:24:48.745400 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.745352    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96"} err="failed to get container status \"a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96\": rpc error: code = NotFound desc = could not find container \"a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96\": container with ID starting with a4619ac8d9f98e8f567842dd2740b210f19e6202b848e09c60cf9863ab9c5b96 not found: ID does not exist"
May 20 14:24:48.745400 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.745365    2554 scope.go:117] "RemoveContainer" containerID="4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677"
May 20 14:24:48.745621 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:24:48.745593    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677\": container with ID starting with 4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677 not found: ID does not exist" containerID="4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677"
May 20 14:24:48.745665 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.745626    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677"} err="failed to get container status \"4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677\": rpc error: code = NotFound desc = could not find container \"4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677\": container with ID starting with 4ef19d4af61dc511ce0a74007e740f62ac839ff3366ef306c9fc7737de21d677 not found: ID does not exist"
May 20 14:24:48.824493 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.824447    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70433e3d-b641-45e6-aa58-1d2a185b098a-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:24:48.824493 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.824485    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70433e3d-b641-45e6-aa58-1d2a185b098a-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:24:48.824493 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:48.824497    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4j6c\" (UniqueName: \"kubernetes.io/projected/70433e3d-b641-45e6-aa58-1d2a185b098a-kube-api-access-s4j6c\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:24:49.028515 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:49.028463    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"]
May 20 14:24:49.031909 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:49.031862    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-55f6fbbcfxwgqm"]
May 20 14:24:50.177472 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:24:50.177434    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" path="/var/lib/kubelet/pods/70433e3d-b641-45e6-aa58-1d2a185b098a/volumes"
May 20 14:25:16.091571 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:25:16.091541    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:25:16.100706 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:25:16.100683    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:26:59.290476 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.290439    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 20 14:26:59.290975 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.290947    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="main"
May 20 14:26:59.290975 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.290961    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="main"
May 20 14:26:59.291058 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.290978    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="tokenizer"
May 20 14:26:59.291058 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.290988    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="tokenizer"
May 20 14:26:59.291058 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.291005    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="storage-initializer"
May 20 14:26:59.291058 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.291011    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="storage-initializer"
May 20 14:26:59.291178 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.291080    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="main"
May 20 14:26:59.291178 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.291095    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="70433e3d-b641-45e6-aa58-1d2a185b098a" containerName="tokenizer"
May 20 14:26:59.294452 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.294415    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.297082 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.297057    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:26:59.297232 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.297058    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-96lmg\""
May 20 14:26:59.297573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.297554    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:26:59.297643 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.297589    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 20 14:26:59.305244 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.305220    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 20 14:26:59.381246 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.381208    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"]
May 20 14:26:59.385733 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.385708    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.388896 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.388825    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-h8d8b\""
May 20 14:26:59.399228 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.399201    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"]
May 20 14:26:59.432367 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432330    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdb2\" (UniqueName: \"kubernetes.io/projected/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kube-api-access-mrdb2\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.432549 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432408    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55565e3-f03a-4bbd-be29-dbe226e8800c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.432549 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.432549 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432529    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.432737 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432554    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.432737 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432583    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.432737 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432601    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.432737 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.432737 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432695    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7p7g\" (UniqueName: \"kubernetes.io/projected/b55565e3-f03a-4bbd-be29-dbe226e8800c-kube-api-access-w7p7g\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.433078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432798    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.433078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432834    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.433078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432893    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.433078 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.432949    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.533849 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.533806    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534045 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.533906    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w7p7g\" (UniqueName: \"kubernetes.io/projected/b55565e3-f03a-4bbd-be29-dbe226e8800c-kube-api-access-w7p7g\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534045 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.533946    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534045 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.533978    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534045 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534026    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534267 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534072    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534267 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534106    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdb2\" (UniqueName: \"kubernetes.io/projected/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kube-api-access-mrdb2\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534267 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534182    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55565e3-f03a-4bbd-be29-dbe226e8800c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534267 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534255    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534293    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534293    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534316    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534322    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534368    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534381    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.534477 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534401    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534890 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534530    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534890 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534545    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.534890 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534681    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.535054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534940    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.535054 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.534976    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.536674 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.536653    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.536907 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.536890    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55565e3-f03a-4bbd-be29-dbe226e8800c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.537058 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.537038    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.541718 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.541660    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdb2\" (UniqueName: \"kubernetes.io/projected/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kube-api-access-mrdb2\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.541905 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.541884    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7p7g\" (UniqueName: \"kubernetes.io/projected/b55565e3-f03a-4bbd-be29-dbe226e8800c-kube-api-access-w7p7g\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.608394 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.608352    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:26:59.697242 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.697080    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:26:59.768768 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.768739    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 20 14:26:59.769998 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:26:59.769955    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55565e3_f03a_4bbd_be29_dbe226e8800c.slice/crio-889e57fe26ddab6373778bc75572c327b062707835806a481e800301a90a8ad9 WatchSource:0}: Error finding container 889e57fe26ddab6373778bc75572c327b062707835806a481e800301a90a8ad9: Status 404 returned error can't find the container with id 889e57fe26ddab6373778bc75572c327b062707835806a481e800301a90a8ad9
May 20 14:26:59.772772 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.772751    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:26:59.876123 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:26:59.876100    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"]
May 20 14:26:59.877446 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:26:59.877410    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6864f9f1_7c2b_4c11_9b32_facb4012bd1f.slice/crio-6ad8be8e6ad2e2496869906d4010ae805e7e787119041bedaf3ff1c6877b3728 WatchSource:0}: Error finding container 6ad8be8e6ad2e2496869906d4010ae805e7e787119041bedaf3ff1c6877b3728: Status 404 returned error can't find the container with id 6ad8be8e6ad2e2496869906d4010ae805e7e787119041bedaf3ff1c6877b3728
May 20 14:27:00.256251 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:00.256196    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerStarted","Data":"5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12"}
May 20 14:27:00.256251 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:00.256255    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerStarted","Data":"6ad8be8e6ad2e2496869906d4010ae805e7e787119041bedaf3ff1c6877b3728"}
May 20 14:27:00.257751 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:00.257717    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"b55565e3-f03a-4bbd-be29-dbe226e8800c","Type":"ContainerStarted","Data":"af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee"}
May 20 14:27:00.257926 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:00.257757    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"b55565e3-f03a-4bbd-be29-dbe226e8800c","Type":"ContainerStarted","Data":"889e57fe26ddab6373778bc75572c327b062707835806a481e800301a90a8ad9"}
May 20 14:27:01.264140 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:01.264108    2554 generic.go:358] "Generic (PLEG): container finished" podID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerID="5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12" exitCode=0
May 20 14:27:01.264615 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:01.264187    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerDied","Data":"5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12"}
May 20 14:27:02.273193 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:02.273152    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerStarted","Data":"7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559"}
May 20 14:27:02.273592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:02.273203    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerStarted","Data":"0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c"}
May 20 14:27:02.273592 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:02.273272    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:27:02.294439 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:02.294352    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" podStartSLOduration=3.294334964 podStartE2EDuration="3.294334964s" podCreationTimestamp="2026-05-20 14:26:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:27:02.292090349 +0000 UTC m=+2206.914815419" watchObservedRunningTime="2026-05-20 14:27:02.294334964 +0000 UTC m=+2206.917060033"
May 20 14:27:09.697770 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:09.697728    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:27:09.697770 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:09.697769    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:27:09.700853 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:09.700812    2554 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:27:10.308994 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:10.308957    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:27:31.314120 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:31.314089    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:27:43.992290 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:43.992191    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"]
May 20 14:27:43.992749 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:43.992508    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6" podUID="4e03b710-ae09-4591-bc03-ce91f5ac3ec1" containerName="manager" containerID="cri-o://2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb" gracePeriod=30
May 20 14:27:44.250648 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.250586    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:27:44.376833 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.376798    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-cert\") pod \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") "
May 20 14:27:44.377024 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.376927    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb2jj\" (UniqueName: \"kubernetes.io/projected/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-kube-api-access-nb2jj\") pod \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\" (UID: \"4e03b710-ae09-4591-bc03-ce91f5ac3ec1\") "
May 20 14:27:44.378853 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.378813    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-cert" (OuterVolumeSpecName: "cert") pod "4e03b710-ae09-4591-bc03-ce91f5ac3ec1" (UID: "4e03b710-ae09-4591-bc03-ce91f5ac3ec1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:27:44.378975 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.378931    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-kube-api-access-nb2jj" (OuterVolumeSpecName: "kube-api-access-nb2jj") pod "4e03b710-ae09-4591-bc03-ce91f5ac3ec1" (UID: "4e03b710-ae09-4591-bc03-ce91f5ac3ec1"). InnerVolumeSpecName "kube-api-access-nb2jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:27:44.452732 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.452693    2554 generic.go:358] "Generic (PLEG): container finished" podID="4e03b710-ae09-4591-bc03-ce91f5ac3ec1" containerID="2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb" exitCode=0
May 20 14:27:44.452970 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.452750    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6" event={"ID":"4e03b710-ae09-4591-bc03-ce91f5ac3ec1","Type":"ContainerDied","Data":"2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb"}
May 20 14:27:44.452970 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.452785    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"
May 20 14:27:44.452970 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.452800    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-58b7b7548c-wgtk6" event={"ID":"4e03b710-ae09-4591-bc03-ce91f5ac3ec1","Type":"ContainerDied","Data":"67aa2035b9d1ac50463c52076640db3738c65d173239b4e81d32b655a93844e4"}
May 20 14:27:44.452970 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.452822    2554 scope.go:117] "RemoveContainer" containerID="2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb"
May 20 14:27:44.463087 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.463064    2554 scope.go:117] "RemoveContainer" containerID="2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb"
May 20 14:27:44.463354 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:27:44.463332    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb\": container with ID starting with 2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb not found: ID does not exist" containerID="2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb"
May 20 14:27:44.463431 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.463367    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb"} err="failed to get container status \"2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb\": rpc error: code = NotFound desc = could not find container \"2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb\": container with ID starting with 2730a402bfed09ac3a531502b30712421c242951a699a6d21e91d5e6233d5adb not found: ID does not exist"
May 20 14:27:44.475688 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.475660    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"]
May 20 14:27:44.478534 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.478507    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2jj\" (UniqueName: \"kubernetes.io/projected/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-kube-api-access-nb2jj\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:27:44.478534 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.478534    2554 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e03b710-ae09-4591-bc03-ce91f5ac3ec1-cert\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:27:44.478980 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:44.478960    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-58b7b7548c-wgtk6"]
May 20 14:27:46.178185 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:46.178155    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e03b710-ae09-4591-bc03-ce91f5ac3ec1" path="/var/lib/kubelet/pods/4e03b710-ae09-4591-bc03-ce91f5ac3ec1/volumes"
May 20 14:27:56.523024 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:56.522771    2554 generic.go:358] "Generic (PLEG): container finished" podID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerID="af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee" exitCode=0
May 20 14:27:56.523024 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:27:56.522997    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"b55565e3-f03a-4bbd-be29-dbe226e8800c","Type":"ContainerDied","Data":"af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee"}
May 20 14:28:24.664700 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:24.664661    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"b55565e3-f03a-4bbd-be29-dbe226e8800c","Type":"ContainerStarted","Data":"02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a"}
May 20 14:28:24.687977 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:24.687908    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podStartSLOduration=58.007265014 podStartE2EDuration="1m25.687887301s" podCreationTimestamp="2026-05-20 14:26:59 +0000 UTC" firstStartedPulling="2026-05-20 14:27:56.524401915 +0000 UTC m=+2261.147126962" lastFinishedPulling="2026-05-20 14:28:24.205024193 +0000 UTC m=+2288.827749249" observedRunningTime="2026-05-20 14:28:24.683600277 +0000 UTC m=+2289.306325364" watchObservedRunningTime="2026-05-20 14:28:24.687887301 +0000 UTC m=+2289.310612395"
May 20 14:28:47.663677 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:47.663642    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"]
May 20 14:28:47.664148 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:47.663988    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="main" containerID="cri-o://0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c" gracePeriod=30
May 20 14:28:47.664148 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:47.664049    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="tokenizer" containerID="cri-o://7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559" gracePeriod=30
May 20 14:28:48.658421 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:48.658381    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 20 14:28:48.658809 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:48.658769    2554 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podUID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerName="main" containerID="cri-o://02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a" gracePeriod=30
May 20 14:28:48.775656 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:48.775620    2554 generic.go:358] "Generic (PLEG): container finished" podID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerID="0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c" exitCode=0
May 20 14:28:48.776061 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:48.775670    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerDied","Data":"0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c"}
May 20 14:28:49.032852 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.032808    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:28:49.145935 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.145891    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kserve-provision-location\") pod \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") "
May 20 14:28:49.146145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.145959    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-tmp\") pod \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") "
May 20 14:28:49.146145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.145993    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrdb2\" (UniqueName: \"kubernetes.io/projected/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kube-api-access-mrdb2\") pod \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") "
May 20 14:28:49.146145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.146035    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tls-certs\") pod \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") "
May 20 14:28:49.146145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.146099    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-cache\") pod \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") "
May 20 14:28:49.146145 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.146138    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-uds\") pod \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\" (UID: \"6864f9f1-7c2b-4c11-9b32-facb4012bd1f\") "
May 20 14:28:49.146490 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.146449    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "6864f9f1-7c2b-4c11-9b32-facb4012bd1f" (UID: "6864f9f1-7c2b-4c11-9b32-facb4012bd1f"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:49.146567 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.146465    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "6864f9f1-7c2b-4c11-9b32-facb4012bd1f" (UID: "6864f9f1-7c2b-4c11-9b32-facb4012bd1f"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:49.146643 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.146584    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6864f9f1-7c2b-4c11-9b32-facb4012bd1f" (UID: "6864f9f1-7c2b-4c11-9b32-facb4012bd1f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:49.147011 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.146984    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6864f9f1-7c2b-4c11-9b32-facb4012bd1f" (UID: "6864f9f1-7c2b-4c11-9b32-facb4012bd1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:49.148274 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.148247    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6864f9f1-7c2b-4c11-9b32-facb4012bd1f" (UID: "6864f9f1-7c2b-4c11-9b32-facb4012bd1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:28:49.148339 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.148269    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kube-api-access-mrdb2" (OuterVolumeSpecName: "kube-api-access-mrdb2") pod "6864f9f1-7c2b-4c11-9b32-facb4012bd1f" (UID: "6864f9f1-7c2b-4c11-9b32-facb4012bd1f"). InnerVolumeSpecName "kube-api-access-mrdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:28:49.247942 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.247857    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:49.247942 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.247890    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-uds\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:49.247942 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.247899    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:49.247942 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.247910    2554 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tokenizer-tmp\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:49.247942 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.247921    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrdb2\" (UniqueName: \"kubernetes.io/projected/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-kube-api-access-mrdb2\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:49.247942 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.247930    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6864f9f1-7c2b-4c11-9b32-facb4012bd1f-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:49.782768 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.782734    2554 generic.go:358] "Generic (PLEG): container finished" podID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerID="7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559" exitCode=0
May 20 14:28:49.783253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.782808    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerDied","Data":"7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559"}
May 20 14:28:49.783253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.782836    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"
May 20 14:28:49.783253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.782873    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh" event={"ID":"6864f9f1-7c2b-4c11-9b32-facb4012bd1f","Type":"ContainerDied","Data":"6ad8be8e6ad2e2496869906d4010ae805e7e787119041bedaf3ff1c6877b3728"}
May 20 14:28:49.783253 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.782897    2554 scope.go:117] "RemoveContainer" containerID="7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559"
May 20 14:28:49.802010 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.801981    2554 scope.go:117] "RemoveContainer" containerID="0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c"
May 20 14:28:49.811549 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.811525    2554 scope.go:117] "RemoveContainer" containerID="5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12"
May 20 14:28:49.814000 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.813972    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"]
May 20 14:28:49.817013 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.816988    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-schegkglh"]
May 20 14:28:49.822676 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.822647    2554 scope.go:117] "RemoveContainer" containerID="7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559"
May 20 14:28:49.823052 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:28:49.823031    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559\": container with ID starting with 7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559 not found: ID does not exist" containerID="7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559"
May 20 14:28:49.823115 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.823063    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559"} err="failed to get container status \"7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559\": rpc error: code = NotFound desc = could not find container \"7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559\": container with ID starting with 7740792bfb7fedf62606e497141ad07dcf49e8be78f5d253486bad1c1544b559 not found: ID does not exist"
May 20 14:28:49.823115 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.823084    2554 scope.go:117] "RemoveContainer" containerID="0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c"
May 20 14:28:49.823353 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:28:49.823330    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c\": container with ID starting with 0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c not found: ID does not exist" containerID="0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c"
May 20 14:28:49.823401 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.823360    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c"} err="failed to get container status \"0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c\": rpc error: code = NotFound desc = could not find container \"0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c\": container with ID starting with 0f614bd4c9f20f650b2e248a12781e7941d0122d791697c9f597d33cdf62110c not found: ID does not exist"
May 20 14:28:49.823401 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.823381    2554 scope.go:117] "RemoveContainer" containerID="5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12"
May 20 14:28:49.823614 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:28:49.823597    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12\": container with ID starting with 5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12 not found: ID does not exist" containerID="5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12"
May 20 14:28:49.823656 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:49.823618    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12"} err="failed to get container status \"5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12\": rpc error: code = NotFound desc = could not find container \"5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12\": container with ID starting with 5f2a8dea37b6882f835c914e2308ccc2691c10a84a14e72790145c701c129c12 not found: ID does not exist"
May 20 14:28:50.177636 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.177603    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" path="/var/lib/kubelet/pods/6864f9f1-7c2b-4c11-9b32-facb4012bd1f/volumes"
May 20 14:28:50.719467 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.719397    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_b55565e3-f03a-4bbd-be29-dbe226e8800c/main/0.log"
May 20 14:28:50.719780 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.719762    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:28:50.789115 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.789090    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1_b55565e3-f03a-4bbd-be29-dbe226e8800c/main/0.log"
May 20 14:28:50.789528 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.789391    2554 generic.go:358] "Generic (PLEG): container finished" podID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerID="02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a" exitCode=130
May 20 14:28:50.789528 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.789455    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"b55565e3-f03a-4bbd-be29-dbe226e8800c","Type":"ContainerDied","Data":"02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a"}
May 20 14:28:50.789528 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.789465    2554 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"
May 20 14:28:50.789528 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.789481    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"b55565e3-f03a-4bbd-be29-dbe226e8800c","Type":"ContainerDied","Data":"889e57fe26ddab6373778bc75572c327b062707835806a481e800301a90a8ad9"}
May 20 14:28:50.789528 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.789497    2554 scope.go:117] "RemoveContainer" containerID="02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a"
May 20 14:28:50.798469 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.798445    2554 scope.go:117] "RemoveContainer" containerID="af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee"
May 20 14:28:50.860420 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.860395    2554 scope.go:117] "RemoveContainer" containerID="02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a"
May 20 14:28:50.860737 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:28:50.860716    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a\": container with ID starting with 02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a not found: ID does not exist" containerID="02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a"
May 20 14:28:50.860786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.860746    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a"} err="failed to get container status \"02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a\": rpc error: code = NotFound desc = could not find container \"02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a\": container with ID starting with 02dabc3619bc2b99473ea269a4cf206ab1b7d095bda29831c9de027e1e13f73a not found: ID does not exist"
May 20 14:28:50.860786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.860766    2554 scope.go:117] "RemoveContainer" containerID="af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee"
May 20 14:28:50.861025 ip-10-0-141-251 kubenswrapper[2554]: E0520 14:28:50.861004    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee\": container with ID starting with af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee not found: ID does not exist" containerID="af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee"
May 20 14:28:50.861089 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.861040    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee"} err="failed to get container status \"af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee\": rpc error: code = NotFound desc = could not find container \"af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee\": container with ID starting with af88f216236c0a43b7888856f52f0864ed6b497d41839d34578a0da02de693ee not found: ID does not exist"
May 20 14:28:50.864369 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864348    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7p7g\" (UniqueName: \"kubernetes.io/projected/b55565e3-f03a-4bbd-be29-dbe226e8800c-kube-api-access-w7p7g\") pod \"b55565e3-f03a-4bbd-be29-dbe226e8800c\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") "
May 20 14:28:50.864435 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864405    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-home\") pod \"b55565e3-f03a-4bbd-be29-dbe226e8800c\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") "
May 20 14:28:50.864472 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864449    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55565e3-f03a-4bbd-be29-dbe226e8800c-tls-certs\") pod \"b55565e3-f03a-4bbd-be29-dbe226e8800c\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") "
May 20 14:28:50.864507 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864489    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-tmp-dir\") pod \"b55565e3-f03a-4bbd-be29-dbe226e8800c\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") "
May 20 14:28:50.864541 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864521    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-model-cache\") pod \"b55565e3-f03a-4bbd-be29-dbe226e8800c\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") "
May 20 14:28:50.864584 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864544    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-dshm\") pod \"b55565e3-f03a-4bbd-be29-dbe226e8800c\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") "
May 20 14:28:50.864635 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864594    2554 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-kserve-provision-location\") pod \"b55565e3-f03a-4bbd-be29-dbe226e8800c\" (UID: \"b55565e3-f03a-4bbd-be29-dbe226e8800c\") "
May 20 14:28:50.864755 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864729    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-home" (OuterVolumeSpecName: "home") pod "b55565e3-f03a-4bbd-be29-dbe226e8800c" (UID: "b55565e3-f03a-4bbd-be29-dbe226e8800c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:50.864947 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864873    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b55565e3-f03a-4bbd-be29-dbe226e8800c" (UID: "b55565e3-f03a-4bbd-be29-dbe226e8800c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:50.865077 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864964    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b55565e3-f03a-4bbd-be29-dbe226e8800c" (UID: "b55565e3-f03a-4bbd-be29-dbe226e8800c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:50.865077 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.864992    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-home\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:50.865077 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.865009    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-model-cache\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:50.866679 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.866657    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b55565e3-f03a-4bbd-be29-dbe226e8800c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b55565e3-f03a-4bbd-be29-dbe226e8800c" (UID: "b55565e3-f03a-4bbd-be29-dbe226e8800c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:28:50.867090 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.867065    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-dshm" (OuterVolumeSpecName: "dshm") pod "b55565e3-f03a-4bbd-be29-dbe226e8800c" (UID: "b55565e3-f03a-4bbd-be29-dbe226e8800c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:50.867143 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.867109    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55565e3-f03a-4bbd-be29-dbe226e8800c-kube-api-access-w7p7g" (OuterVolumeSpecName: "kube-api-access-w7p7g") pod "b55565e3-f03a-4bbd-be29-dbe226e8800c" (UID: "b55565e3-f03a-4bbd-be29-dbe226e8800c"). InnerVolumeSpecName "kube-api-access-w7p7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:28:50.923236 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.923191    2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b55565e3-f03a-4bbd-be29-dbe226e8800c" (UID: "b55565e3-f03a-4bbd-be29-dbe226e8800c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:50.966456 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.966421    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7p7g\" (UniqueName: \"kubernetes.io/projected/b55565e3-f03a-4bbd-be29-dbe226e8800c-kube-api-access-w7p7g\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:50.966456 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.966455    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b55565e3-f03a-4bbd-be29-dbe226e8800c-tls-certs\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:50.966456 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.966464    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-tmp-dir\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:50.966678 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.966474    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-dshm\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:50.966678 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:50.966483    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b55565e3-f03a-4bbd-be29-dbe226e8800c-kserve-provision-location\") on node \"ip-10-0-141-251.ec2.internal\" DevicePath \"\""
May 20 14:28:51.114850 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:51.114799    2554 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 20 14:28:51.117415 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:51.117390    2554 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"]
May 20 14:28:52.177644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:28:52.177605    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b55565e3-f03a-4bbd-be29-dbe226e8800c" path="/var/lib/kubelet/pods/b55565e3-f03a-4bbd-be29-dbe226e8800c/volumes"
May 20 14:29:03.943451 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:03.943414    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:03.975150 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:03.975109    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:05.006372 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:05.006343    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:05.024038 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:05.023986    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:06.012147 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:06.012117    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:06.026892 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:06.026823    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:06.990138 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:06.990109    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:07.004190 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:07.004160    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:07.937082 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:07.937049    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:07.952515 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:07.952485    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:09.017433 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:09.017375    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:09.031581 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:09.031552    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:09.986442 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:09.986410    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:10.000788 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:10.000756    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:10.957604 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:10.957572    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:10.971065 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:10.971044    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:11.926435 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:11.926402    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:11.941826 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:11.941789    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:12.893368 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:12.893326    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:12.909723 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:12.909693    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:13.865062 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:13.865033    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:13.881598 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:13.881570    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:14.844334 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:14.844306    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:14.860702 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:14.860671    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:15.837807 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:15.837753    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:15.852644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:15.852606    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:16.844938 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:16.844903    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-7b9cf87876-d6vqg_89d1d61f-8fdc-4183-9de2-e78fe30178f3/istio-proxy/0.log"
May 20 14:29:16.861205 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:16.861151    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-x7rzn_60bf9e3c-3d11-4364-817a-33f00307fc7b/istio-proxy/0.log"
May 20 14:29:18.001490 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:18.001455    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-84b84d7ccf-g66f7_f3240ff9-7b7b-4cbe-a8de-841b7f425c80/istio-proxy/0.log"
May 20 14:29:18.883875 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:18.883829    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-84b84d7ccf-g66f7_f3240ff9-7b7b-4cbe-a8de-841b7f425c80/istio-proxy/0.log"
May 20 14:29:19.703740 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:19.703702    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-84b84d7ccf-g66f7_f3240ff9-7b7b-4cbe-a8de-841b7f425c80/istio-proxy/0.log"
May 20 14:29:20.443573 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:20.443540    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-k8v9v_9901f55d-73bd-4c37-b389-05ba74718cc0/manager/0.log"
May 20 14:29:20.510039 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:20.510008    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-z2bt4_dab24b37-b09d-4300-945b-04bfa02377f1/manager/0.log"
May 20 14:29:20.523441 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:20.523413    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2nm2m_42ff960b-532e-4f74-ae08-5277b8808aac/limitador/0.log"
May 20 14:29:20.539706 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:20.539656    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-qqh27_86e298cc-dd62-489e-9cb2-180bb03d395e/manager/0.log"
May 20 14:29:21.331345 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:21.331312    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-k8v9v_9901f55d-73bd-4c37-b389-05ba74718cc0/manager/0.log"
May 20 14:29:21.401979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:21.401946    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-z2bt4_dab24b37-b09d-4300-945b-04bfa02377f1/manager/0.log"
May 20 14:29:21.413998 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:21.413968    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2nm2m_42ff960b-532e-4f74-ae08-5277b8808aac/limitador/0.log"
May 20 14:29:21.436877 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:21.436830    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-qqh27_86e298cc-dd62-489e-9cb2-180bb03d395e/manager/0.log"
May 20 14:29:22.257967 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:22.257879    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-k8v9v_9901f55d-73bd-4c37-b389-05ba74718cc0/manager/0.log"
May 20 14:29:22.334385 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:22.334351    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-z2bt4_dab24b37-b09d-4300-945b-04bfa02377f1/manager/0.log"
May 20 14:29:22.359126 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:22.359096    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2nm2m_42ff960b-532e-4f74-ae08-5277b8808aac/limitador/0.log"
May 20 14:29:22.377719 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:22.377693    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-qqh27_86e298cc-dd62-489e-9cb2-180bb03d395e/manager/0.log"
May 20 14:29:23.184998 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:23.184967    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-k8v9v_9901f55d-73bd-4c37-b389-05ba74718cc0/manager/0.log"
May 20 14:29:23.255358 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:23.255320    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-z2bt4_dab24b37-b09d-4300-945b-04bfa02377f1/manager/0.log"
May 20 14:29:23.268153 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:23.268123    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2nm2m_42ff960b-532e-4f74-ae08-5277b8808aac/limitador/0.log"
May 20 14:29:23.287051 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:23.287022    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-qqh27_86e298cc-dd62-489e-9cb2-180bb03d395e/manager/0.log"
May 20 14:29:24.073954 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:24.073926    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-k8v9v_9901f55d-73bd-4c37-b389-05ba74718cc0/manager/0.log"
May 20 14:29:24.142368 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:24.142321    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-z2bt4_dab24b37-b09d-4300-945b-04bfa02377f1/manager/0.log"
May 20 14:29:24.150715 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:24.150683    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2nm2m_42ff960b-532e-4f74-ae08-5277b8808aac/limitador/0.log"
May 20 14:29:24.165155 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:24.165126    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-qqh27_86e298cc-dd62-489e-9cb2-180bb03d395e/manager/0.log"
May 20 14:29:26.425437 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.425396    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cpmvk/must-gather-w2jdj"]
May 20 14:29:26.426040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426018    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="tokenizer"
May 20 14:29:26.426040 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426042    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="tokenizer"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426056    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="main"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426062    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="main"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426071    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerName="storage-initializer"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426078    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerName="storage-initializer"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426088    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="storage-initializer"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426093    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="storage-initializer"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426109    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e03b710-ae09-4591-bc03-ce91f5ac3ec1" containerName="manager"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426114    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e03b710-ae09-4591-bc03-ce91f5ac3ec1" containerName="manager"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426122    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerName="main"
May 20 14:29:26.426132 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426127    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerName="main"
May 20 14:29:26.426440 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426192    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="tokenizer"
May 20 14:29:26.426440 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426201    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e03b710-ae09-4591-bc03-ce91f5ac3ec1" containerName="manager"
May 20 14:29:26.426440 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426208    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="b55565e3-f03a-4bbd-be29-dbe226e8800c" containerName="main"
May 20 14:29:26.426440 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.426215    2554 memory_manager.go:356] "RemoveStaleState removing state" podUID="6864f9f1-7c2b-4c11-9b32-facb4012bd1f" containerName="main"
May 20 14:29:26.429137 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.429121    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.431679 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.431656    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpmvk\"/\"kube-root-ca.crt\""
May 20 14:29:26.432422 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.432402    2554 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpmvk\"/\"openshift-service-ca.crt\""
May 20 14:29:26.432539 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.432409    2554 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cpmvk\"/\"default-dockercfg-rh55f\""
May 20 14:29:26.437147 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.437032    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/must-gather-w2jdj"]
May 20 14:29:26.492237 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.492199    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgxc\" (UniqueName: \"kubernetes.io/projected/be2e5c3c-c0c0-4383-b5fe-5b766181ee62-kube-api-access-ttgxc\") pod \"must-gather-w2jdj\" (UID: \"be2e5c3c-c0c0-4383-b5fe-5b766181ee62\") " pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.492416 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.492331    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/be2e5c3c-c0c0-4383-b5fe-5b766181ee62-must-gather-output\") pod \"must-gather-w2jdj\" (UID: \"be2e5c3c-c0c0-4383-b5fe-5b766181ee62\") " pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.593120 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.593083    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/be2e5c3c-c0c0-4383-b5fe-5b766181ee62-must-gather-output\") pod \"must-gather-w2jdj\" (UID: \"be2e5c3c-c0c0-4383-b5fe-5b766181ee62\") " pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.593320 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.593159    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgxc\" (UniqueName: \"kubernetes.io/projected/be2e5c3c-c0c0-4383-b5fe-5b766181ee62-kube-api-access-ttgxc\") pod \"must-gather-w2jdj\" (UID: \"be2e5c3c-c0c0-4383-b5fe-5b766181ee62\") " pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.593452 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.593431    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/be2e5c3c-c0c0-4383-b5fe-5b766181ee62-must-gather-output\") pod \"must-gather-w2jdj\" (UID: \"be2e5c3c-c0c0-4383-b5fe-5b766181ee62\") " pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.601636 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.601605    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgxc\" (UniqueName: \"kubernetes.io/projected/be2e5c3c-c0c0-4383-b5fe-5b766181ee62-kube-api-access-ttgxc\") pod \"must-gather-w2jdj\" (UID: \"be2e5c3c-c0c0-4383-b5fe-5b766181ee62\") " pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.741009 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.740921    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/must-gather-w2jdj"
May 20 14:29:26.880400 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.880375    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/must-gather-w2jdj"]
May 20 14:29:26.882151 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:29:26.882123    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe2e5c3c_c0c0_4383_b5fe_5b766181ee62.slice/crio-1751ddb0ee1a41ba1f19152bc92c0bc1877d06f962ef888c3db906787cdda327 WatchSource:0}: Error finding container 1751ddb0ee1a41ba1f19152bc92c0bc1877d06f962ef888c3db906787cdda327: Status 404 returned error can't find the container with id 1751ddb0ee1a41ba1f19152bc92c0bc1877d06f962ef888c3db906787cdda327
May 20 14:29:26.953389 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:26.953345    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/must-gather-w2jdj" event={"ID":"be2e5c3c-c0c0-4383-b5fe-5b766181ee62","Type":"ContainerStarted","Data":"1751ddb0ee1a41ba1f19152bc92c0bc1877d06f962ef888c3db906787cdda327"}
May 20 14:29:27.962140 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:27.961899    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/must-gather-w2jdj" event={"ID":"be2e5c3c-c0c0-4383-b5fe-5b766181ee62","Type":"ContainerStarted","Data":"3181cf761bb1dbc07d144deb498f748bbf62ba43ac015eb1c667a47cb707768c"}
May 20 14:29:27.962140 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:27.961950    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/must-gather-w2jdj" event={"ID":"be2e5c3c-c0c0-4383-b5fe-5b766181ee62","Type":"ContainerStarted","Data":"5667ce548da9816be3f4c4dd5e7bf02898b39116f73485d2ae021f7ef8953570"}
May 20 14:29:27.979440 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:27.979388    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-cpmvk/must-gather-w2jdj" podStartSLOduration=1.16988228 podStartE2EDuration="1.979370214s" podCreationTimestamp="2026-05-20 14:29:26 +0000 UTC" firstStartedPulling="2026-05-20 14:29:26.884051403 +0000 UTC m=+2351.506776453" lastFinishedPulling="2026-05-20 14:29:27.693539338 +0000 UTC m=+2352.316264387" observedRunningTime="2026-05-20 14:29:27.977154508 +0000 UTC m=+2352.599879576" watchObservedRunningTime="2026-05-20 14:29:27.979370214 +0000 UTC m=+2352.602095281"
May 20 14:29:29.281584 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:29.281533    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-smc4h_5d5a03ba-819e-4e8c-b589-37e651c8f956/global-pull-secret-syncer/0.log"
May 20 14:29:29.351196 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:29.351156    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k2wv5_1b32bac4-4900-4fb5-bfe7-191ce7aab365/konnectivity-agent/0.log"
May 20 14:29:29.458709 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:29.458672    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-251.ec2.internal_0f62113004acf442450518e8b488b56e/haproxy/0.log"
May 20 14:29:33.780812 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:33.780781    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-k8v9v_9901f55d-73bd-4c37-b389-05ba74718cc0/manager/0.log"
May 20 14:29:33.862019 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:33.861985    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-z2bt4_dab24b37-b09d-4300-945b-04bfa02377f1/manager/0.log"
May 20 14:29:33.887080 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:33.887052    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2nm2m_42ff960b-532e-4f74-ae08-5277b8808aac/limitador/0.log"
May 20 14:29:33.919653 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:33.919620    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-qqh27_86e298cc-dd62-489e-9cb2-180bb03d395e/manager/0.log"
May 20 14:29:35.245431 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.245386    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-9w95s_c1d0459a-6c74-4baa-80eb-53e8e1039f03/monitoring-plugin/0.log"
May 20 14:29:35.431098 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.431070    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xp85s_558b53d4-03e9-4e11-a495-b63b06d7c27b/node-exporter/0.log"
May 20 14:29:35.451901 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.451877    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xp85s_558b53d4-03e9-4e11-a495-b63b06d7c27b/kube-rbac-proxy/0.log"
May 20 14:29:35.476857 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.476813    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xp85s_558b53d4-03e9-4e11-a495-b63b06d7c27b/init-textfile/0.log"
May 20 14:29:35.817076 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.817044    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-lb2j2_e9938334-d9bc-489a-ba96-685498e8af9c/prometheus-operator-admission-webhook/0.log"
May 20 14:29:35.926308 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.926277    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dd5b6b687-2d944_8636cff6-00f2-452a-b5aa-5a2c19fd5df9/thanos-query/0.log"
May 20 14:29:35.948895 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.948796    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dd5b6b687-2d944_8636cff6-00f2-452a-b5aa-5a2c19fd5df9/kube-rbac-proxy-web/0.log"
May 20 14:29:35.968686 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.968650    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dd5b6b687-2d944_8636cff6-00f2-452a-b5aa-5a2c19fd5df9/kube-rbac-proxy/0.log"
May 20 14:29:35.989703 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:35.989669    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dd5b6b687-2d944_8636cff6-00f2-452a-b5aa-5a2c19fd5df9/prom-label-proxy/0.log"
May 20 14:29:36.014258 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:36.014226    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dd5b6b687-2d944_8636cff6-00f2-452a-b5aa-5a2c19fd5df9/kube-rbac-proxy-rules/0.log"
May 20 14:29:36.037013 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:36.036964    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dd5b6b687-2d944_8636cff6-00f2-452a-b5aa-5a2c19fd5df9/kube-rbac-proxy-metrics/0.log"
May 20 14:29:37.199880 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:37.199822    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-rcxc9_7a65ae7e-362a-42cd-a0d2-8275da7461f9/networking-console-plugin/0.log"
May 20 14:29:37.732550 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:37.732520    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/1.log"
May 20 14:29:37.745484 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:37.745436    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-kmw87_75488ef6-b873-4e91-977f-d87376a79f00/console-operator/2.log"
May 20 14:29:38.157675 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.157640    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-774b44c455-ffzwg_24e8b27d-13fa-4a0f-9220-63e048705b7d/console/0.log"
May 20 14:29:38.578888 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.578788    2554 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"]
May 20 14:29:38.588743 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.588710    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.592815 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.592722    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"]
May 20 14:29:38.723740 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.723692    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-podres\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.723977 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.723751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-lib-modules\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.723977 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.723792    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-proc\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.723977 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.723824    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ztd\" (UniqueName: \"kubernetes.io/projected/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-kube-api-access-f4ztd\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.723977 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.723923    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-sys\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.824715 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.824675    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-sys\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.824951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.824764    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-podres\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.824951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.824782    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-lib-modules\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.824951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.824805    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-proc\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.824951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.824814    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-sys\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.824951 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.824827    2554 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ztd\" (UniqueName: \"kubernetes.io/projected/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-kube-api-access-f4ztd\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.825162 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.824992    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-podres\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.825162 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.825004    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-proc\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.825162 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.825127    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-lib-modules\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.833621 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.833538    2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ztd\" (UniqueName: \"kubernetes.io/projected/c1dda2d2-ce0d-4add-9fa0-3f62b488cb37-kube-api-access-f4ztd\") pod \"perf-node-gather-daemonset-4wf4c\" (UID: \"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:38.904719 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:38.904682    2554 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:39.078792 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:39.078744    2554 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"]
May 20 14:29:39.079332 ip-10-0-141-251 kubenswrapper[2554]: W0520 14:29:39.079299    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc1dda2d2_ce0d_4add_9fa0_3f62b488cb37.slice/crio-0e13d3208d3c28fc4ee257c7f750514a11e502490c1a66067ddb1613e0efe57e WatchSource:0}: Error finding container 0e13d3208d3c28fc4ee257c7f750514a11e502490c1a66067ddb1613e0efe57e: Status 404 returned error can't find the container with id 0e13d3208d3c28fc4ee257c7f750514a11e502490c1a66067ddb1613e0efe57e
May 20 14:29:39.433566 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:39.433520    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtp2c_17a72c68-134e-4e0e-a8e7-e278d52ef7dc/dns/0.log"
May 20 14:29:39.451828 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:39.451803    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtp2c_17a72c68-134e-4e0e-a8e7-e278d52ef7dc/kube-rbac-proxy/0.log"
May 20 14:29:39.512786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:39.512758    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7cqn4_f817fca4-539b-4c30-8739-dd4417db0223/dns-node-resolver/0.log"
May 20 14:29:40.029455 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:40.029414    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c" event={"ID":"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37","Type":"ContainerStarted","Data":"0741f9ac7fdbc6391953e6fa7d8a3b19add73d14788770b134eabbc0985a3b5b"}
May 20 14:29:40.029884 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:40.029555    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:40.029884 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:40.029579    2554 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c" event={"ID":"c1dda2d2-ce0d-4add-9fa0-3f62b488cb37","Type":"ContainerStarted","Data":"0e13d3208d3c28fc4ee257c7f750514a11e502490c1a66067ddb1613e0efe57e"}
May 20 14:29:40.046164 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:40.046116    2554 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c" podStartSLOduration=2.046098776 podStartE2EDuration="2.046098776s" podCreationTimestamp="2026-05-20 14:29:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:29:40.043154374 +0000 UTC m=+2364.665879455" watchObservedRunningTime="2026-05-20 14:29:40.046098776 +0000 UTC m=+2364.668823843"
May 20 14:29:40.049100 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:40.049032    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fg6nm_2790ca0e-0d38-4d36-94bd-f9d07e2aba29/node-ca/0.log"
May 20 14:29:40.920533 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:40.920480    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-84b84d7ccf-g66f7_f3240ff9-7b7b-4cbe-a8de-841b7f425c80/istio-proxy/0.log"
May 20 14:29:41.415387 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:41.415360    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gngpj_22742942-2c3a-4d4b-b354-8cf1476abc4e/serve-healthcheck-canary/0.log"
May 20 14:29:41.821153 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:41.821060    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-8mb69_61e2746b-3239-4211-8013-f40b5b0f7ccc/insights-operator/0.log"
May 20 14:29:41.821421 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:41.821401    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-8mb69_61e2746b-3239-4211-8013-f40b5b0f7ccc/insights-operator/1.log"
May 20 14:29:41.840554 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:41.840517    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f49c5_54a00923-9d3c-4d5c-98cf-adbbb391b2ca/kube-rbac-proxy/0.log"
May 20 14:29:41.859097 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:41.859068    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f49c5_54a00923-9d3c-4d5c-98cf-adbbb391b2ca/exporter/0.log"
May 20 14:29:41.880803 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:41.880771    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f49c5_54a00923-9d3c-4d5c-98cf-adbbb391b2ca/extractor/0.log"
May 20 14:29:45.043677 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:45.043638    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-867cffc775-ndt9f_810eb18b-874f-42d5-bb41-ebc13e8df7c2/manager/0.log"
May 20 14:29:45.088565 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:45.088529    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-zvn8b_e3570305-84ea-4bb8-a857-370dded78754/server/0.log"
May 20 14:29:45.502896 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:45.502864    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-jzmm8_b7989cc1-fe4b-458b-9870-3b2ea44fcc94/manager/0.log"
May 20 14:29:45.546389 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:45.546356    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5vxwn_6bccc336-51aa-4290-a0ea-71be31b86593/s3-init/0.log"
May 20 14:29:45.562499 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:45.562470    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5vxwn_6bccc336-51aa-4290-a0ea-71be31b86593/download-hf-model/0.log"
May 20 14:29:45.589374 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:45.589343    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-5tfqj_2d754395-29c2-40a1-864b-577565d9acb4/seaweedfs/0.log"
May 20 14:29:46.043552 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:46.043519    2554 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-4wf4c"
May 20 14:29:50.155621 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:50.155592    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fq9s5_216fb31e-13ab-4026-a416-7848a037a823/migrator/0.log"
May 20 14:29:50.177265 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:50.177239    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-fq9s5_216fb31e-13ab-4026-a416-7848a037a823/graceful-termination/0.log"
May 20 14:29:50.509824 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:50.509727    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mcdf2_9a06524c-de8f-4f26-8ba4-ae198f93fdbf/kube-storage-version-migrator-operator/1.log"
May 20 14:29:50.511396 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:50.511371    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-mcdf2_9a06524c-de8f-4f26-8ba4-ae198f93fdbf/kube-storage-version-migrator-operator/0.log"
May 20 14:29:51.465606 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.465577    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56jr7_d10bae7d-9cef-4c47-9918-5c9748ff88a2/kube-multus-additional-cni-plugins/0.log"
May 20 14:29:51.484662 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.484633    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56jr7_d10bae7d-9cef-4c47-9918-5c9748ff88a2/egress-router-binary-copy/0.log"
May 20 14:29:51.505225 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.505148    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56jr7_d10bae7d-9cef-4c47-9918-5c9748ff88a2/cni-plugins/0.log"
May 20 14:29:51.526380 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.526347    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56jr7_d10bae7d-9cef-4c47-9918-5c9748ff88a2/bond-cni-plugin/0.log"
May 20 14:29:51.554075 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.554030    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56jr7_d10bae7d-9cef-4c47-9918-5c9748ff88a2/routeoverride-cni/0.log"
May 20 14:29:51.571374 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.571345    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56jr7_d10bae7d-9cef-4c47-9918-5c9748ff88a2/whereabouts-cni-bincopy/0.log"
May 20 14:29:51.589285 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.589250    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-56jr7_d10bae7d-9cef-4c47-9918-5c9748ff88a2/whereabouts-cni/0.log"
May 20 14:29:51.949979 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.949942    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n9nk9_575a7241-3e3e-4fbd-a215-e69cb73bf5a4/kube-multus/0.log"
May 20 14:29:51.972140 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.972097    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n4cgv_b8c29338-36d1-4d0d-942e-6b5312426efc/network-metrics-daemon/0.log"
May 20 14:29:51.988345 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:51.988298    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n4cgv_b8c29338-36d1-4d0d-942e-6b5312426efc/kube-rbac-proxy/0.log"
May 20 14:29:53.461147 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.461114    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/ovn-controller/0.log"
May 20 14:29:53.502995 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.502962    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/ovn-acl-logging/0.log"
May 20 14:29:53.526422 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.526393    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/kube-rbac-proxy-node/0.log"
May 20 14:29:53.548342 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.548301    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/kube-rbac-proxy-ovn-metrics/0.log"
May 20 14:29:53.563207 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.563181    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/northd/0.log"
May 20 14:29:53.582724 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.582694    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/nbdb/0.log"
May 20 14:29:53.602786 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.602748    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/sbdb/0.log"
May 20 14:29:53.811679 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:53.811600    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xd45j_32db493c-1a8f-4925-ada5-8f0562c64769/ovnkube-controller/0.log"
May 20 14:29:54.810644 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:54.810593    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-h5p4t_e499f464-38b8-49aa-977c-c3b5054cd843/check-endpoints/0.log"
May 20 14:29:54.874605 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:54.874574    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-z9p8j_77da8e32-a72e-4ec4-96dc-97db3ede4765/network-check-target-container/0.log"
May 20 14:29:55.819996 ip-10-0-141-251 kubenswrapper[2554]: I0520 14:29:55.819967    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5tbmv_b78bf5cb-cabe-4493-9b8c-c3b0be0f493e/iptables-alerter/0.log"