Sep 05 02:15:48.387296 ip-10-0-138-91 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 05 02:15:48.387307 ip-10-0-138-91 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 05 02:15:48.387314 ip-10-0-138-91 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 05 02:15:48.387556 ip-10-0-138-91 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 05 02:15:59.545045 ip-10-0-138-91 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 05 02:15:59.545060 ip-10-0-138-91 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3357c68ab0fc4f33afccf550703974cc --
Sep 05 02:17:35.918938 ip-10-0-138-91 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 02:17:36.298105 ip-10-0-138-91 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:36.298105 ip-10-0-138-91 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 02:17:36.298105 ip-10-0-138-91 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:36.298105 ip-10-0-138-91 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 02:17:36.298105 ip-10-0-138-91 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 02:17:36.300125 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.300048    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 02:17:36.304039 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304024    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.304039 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304040    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304043    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304047    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304050    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304053    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304056    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304059    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304062    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304065    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304068    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304070    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304073    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304083    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304086    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304089    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304092    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304094    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304097    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304100    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304102    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304105    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.304102 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304108    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304111    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304114    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304117    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304120    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304122    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304125    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304127    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304130    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304133    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304135    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304140    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304144    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304147    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304151    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304154    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304158    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304161    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304164    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.304590 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304167    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304170    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304173    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304175    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304178    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304180    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304183    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304185    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304188    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304191    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304193    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304196    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304198    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304201    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304203    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304206    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304209    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304212    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304214    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.305067 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304217    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304220    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304225    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304228    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304231    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304233    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304236    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304239    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304241    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304244    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304246    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304250    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304252    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304255    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304258    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304260    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304263    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304266    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304268    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304271    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.305537 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304273    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304276    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304279    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304281    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304284    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304286    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304289    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304292    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304295    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304297    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304300    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304303    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304305    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304308    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304311    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304314    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304316    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304319    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304321    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.304997    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.306069 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305004    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305007    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305011    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305013    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305016    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305019    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305022    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305024    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305027    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305030    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305032    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305035    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305037    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305040    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305043    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305045    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305048    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305051    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305054    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305056    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.306551 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305059    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305062    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305064    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305067    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305070    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305073    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305076    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305078    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305081    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305083    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305086    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305088    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305091    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305094    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305096    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305099    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305101    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305112    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305115    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305118    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.307071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305120    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305123    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305125    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305128    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305131    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305136    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305139    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305143    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305147    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305151    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305154    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305157    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305159    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305162    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305165    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305168    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305170    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305173    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305176    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.307559 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305178    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305181    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305183    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305186    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305188    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305191    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305193    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305196    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305198    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305202    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305206    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305209    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305212    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305216    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305219    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305222    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305225    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305228    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305230    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305233    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.308028 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305236    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305239    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305242    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305244    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305247    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305250    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305252    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305255    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305258    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305260    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305263    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305265    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305268    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305270    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305273    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305276    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305278    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305281    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305283    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.308518 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305631    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305636    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305638    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305641    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305644    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305647    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305650    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305653    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305655    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305671    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305674    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305677    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305680    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305683    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305686    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305689    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305691    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305694    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305696    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305699    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.308987 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305701    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305704    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305707    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305709    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305712    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305714    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305717    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305719    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305722    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305725    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305727    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305730    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305732    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305735    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305737    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305740    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305742    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305745    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305748    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305750    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.309467 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305753    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305755    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305757    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305760    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305763    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305766    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305769    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305771    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305774    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305777    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305779    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305782    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305784    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305787    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305790    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305792    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305796    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305798    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305801    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305804    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.309977 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305806    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305809    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305811    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305814    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305816    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305819    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305821    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305823    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305826    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305828    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305833    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305836    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305839    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305842    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305845    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305850    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305853    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305855    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305859    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.310472 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305862    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305865    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305869    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305873    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305875    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305879    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305881    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305884    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305887    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305890    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305892    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305895    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305897    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305900    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305902    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305905    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305907    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305910    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305912    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.305915    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.310980 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.305987    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306018    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306025    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306029    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306033    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306036    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306040    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306044    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306049    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306052    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306055    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306058    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306062    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306064    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306067    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306070    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306073    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306076    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306080    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306083    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306086    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306089    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306092    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306096    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 02:17:36.311474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306098    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306101    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306104    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306107    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306110    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306113    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306116    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306119    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306123    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306126    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306128    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306131    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306134    2579 flags.go:64] FLAG: --enable-server="true"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306137    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306142    2579 flags.go:64] FLAG: --event-burst="100"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306145    2579 flags.go:64] FLAG: --event-qps="50"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306152    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306155    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306159    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306162    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306165    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306168    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306171    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306174    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306177    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 02:17:36.312056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306180    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306183    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306186    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306188    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306191    2579 flags.go:64] FLAG: --feature-gates=""
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306199    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306202    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306205    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306208    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306211    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306214    2579 flags.go:64] FLAG: --help="false"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306217    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306220    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306223    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306226    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306230    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306233    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306236    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306238    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306241    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306244    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306247    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306251    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306255    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 05 02:17:36.312685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306258    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306261    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306264    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306267    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306270    2579 flags.go:64] FLAG: --lock-file=""
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306273    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306276    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306279    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306284    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306286    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306289    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306292    2579 flags.go:64] FLAG: --logging-format="text"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306295    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306298    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306301    2579 flags.go:64] FLAG: --manifest-url=""
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306303    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306307    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306310    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306314    2579 flags.go:64] FLAG: --max-pods="110"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306317    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306320    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306323    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306326    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306330    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306333    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 02:17:36.313286 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306336    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306342    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306345    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306349    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306352    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306356    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306363    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306366    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306369    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306372    2579 flags.go:64] FLAG: --port="10250"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306375    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306378    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02336534f57c09e4a"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306381    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306384    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306387    2579 flags.go:64] FLAG: --register-node="true"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306390    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306393    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306396    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306399    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306402    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306405    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306408    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306412    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306415    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306418    2579 flags.go:64] FLAG: --runonce="false"
Sep 05 02:17:36.313941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306421    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306423    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306426    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306429    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306432    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306435    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306439    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306442    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306445    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306448    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306451    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306454    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306456    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306461    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306464    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306469    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306472    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306474    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306478    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306481    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306484    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306486    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306489    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306492    2579 flags.go:64] FLAG: --v="2"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306496    2579 flags.go:64] FLAG: --version="false"
Sep 05 02:17:36.314570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306499    2579 flags.go:64] FLAG: --vmodule=""
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306503    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.306506    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306615    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306620    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306624    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306627    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306630    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306633    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306636    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306639    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306643    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306647    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306651    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306654    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306656    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306672    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306675    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306679    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.315201 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306682    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306686    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306690    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306693    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306696    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306699    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306702    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306704    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306707    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306710    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306712    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306715    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306718    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306720    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306723    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306725    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306728    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306730    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306733    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306735    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.315693 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306738    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306741    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306743    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306746    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306748    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306751    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306755    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306757    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306760    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306762    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306765    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306767    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306770    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306773    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306775    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306778    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306781    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306784    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306786    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306789    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.316179 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306792    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306794    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306797    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306800    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306802    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306804    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306807    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306809    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306812    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306815    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306817    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306820    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306822    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306825    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306827    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306830    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306832    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306835    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306839    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306842    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.316763 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306844    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306847    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306849    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306853    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306855    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306858    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306861    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306863    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306866    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306869    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306871    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306874    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306876    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306879    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306881    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306884    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306886    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306889    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306891    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306894    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.317262 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306896    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306899    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.306901    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.307566    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.314477    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.314491    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.314509    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.314514    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314565    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314570    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314575    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314579    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314583    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314586    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.317796 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314588    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314591    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314594    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314596    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314599    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314602    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314604    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314607    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314610    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314613    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314615    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314617    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314620    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314623    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314625    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314628    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314630    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314633    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314635    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314638    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.318144 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314640    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314643    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314646    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314649    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314652    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314655    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314674    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314680    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314683    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314686    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314690    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314692    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314695    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314698    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314701    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314704    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314708    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314711    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314714    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.318631 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314716    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314719    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314722    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314725    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314727    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314730    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314732    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314735    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314737    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314740    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314742    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314745    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314747    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314750    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314752    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314755    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314758    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314762    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314764    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314767    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.319104 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314769    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314772    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314774    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314777    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314779    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314781    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314784    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314786    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314789    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314792    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314794    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314797    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314800    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314802    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314805    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314807    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314810    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314812    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314815    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.319595 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314817    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314820    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314823    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314825    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314828    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314830    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314832    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314835    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314838    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314840    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314843    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314847    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314849    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314852    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314854    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.320146 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.314859    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314979    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314984    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314987    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314990    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314993    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314996    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.314999    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315001    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315004    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315006    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315009    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315011    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315014    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315017    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315019    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315022    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315024    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315027    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315029    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 02:17:36.320757 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315032    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315034    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315037    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315039    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315042    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315045    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315047    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315050    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315052    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315055    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315058    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315061    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315063    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315066    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315068    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315071    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315073    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315076    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315079    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315081    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 02:17:36.321301 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315084    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315086    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315089    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315091    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315093    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315097    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315101    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315103    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315106    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315109    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315111    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315113    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315116    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315118    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315121    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315123    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315126    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315129    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315131    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315134    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 02:17:36.321815 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315137    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315139    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315142    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315145    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315147    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315150    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315154    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315157    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315160    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315163    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315166    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315168    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315171    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315174    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315176    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315179    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315182    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315184    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315186    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 02:17:36.322298 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315189    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315192    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315194    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315197    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315199    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315202    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315205    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315207    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315211    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315213    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315216    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315219    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315221    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315224    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315226    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315229    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315233    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315236    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315239    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315241    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 02:17:36.322776 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:36.315244    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 02:17:36.323259 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.315249    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 02:17:36.323259 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.315335    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 02:17:36.323259 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.321160    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 02:17:36.323259 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.322240    2579 server.go:1019] "Starting client certificate rotation"
Sep 05 02:17:36.323259 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.322339    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:36.323259 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.322378    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 02:17:36.325045 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.325032    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:36.327864 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.327844    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 02:17:36.342058 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.341968    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 05 02:17:36.345327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.345306    2579 log.go:25] "Validated CRI v1 image API"
Sep 05 02:17:36.346612 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.346591    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 02:17:36.349712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.349688    2579 fs.go:135] Filesystem UUIDs: map[617aa6f2-4cc2-4985-9a10-e21dbe3152f6:/dev/nvme0n1p3 77505ff0-c67a-4ede-aa72-b7043dd18901:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 05 02:17:36.349775 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.349710    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 02:17:36.356194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.356048    2579 manager.go:216] Machine: {Timestamp:2026-09-05 02:17:36.353888769 +0000 UTC m=+0.326192636 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102808 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274147ee1753127e3a92c65e9cf110 SystemUUID:ec274147-ee17-5312-7e3a-92c65e9cf110 BootID:3357c68a-b0fc-4f33-afcc-f550703974cc Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c6:4b:03:be:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c6:4b:03:be:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:ce:11:08:ab:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 02:17:36.356194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.356185    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 02:17:36.356305 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.356278    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 02:17:36.358098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.358082    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:36.359569 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.359541    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 02:17:36.359781 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.359578    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 02:17:36.360390 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.360379    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 02:17:36.360433 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.360393    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 02:17:36.360433 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.360400    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:36.361147 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.361136    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 02:17:36.362194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.362185    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 02:17:36.366298 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366281    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:36.366405 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366395    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 02:17:36.366440 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366423    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 02:17:36.366440 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366433    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 02:17:36.366494 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366443    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 02:17:36.366494 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366452    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 05 02:17:36.366494 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366459    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 02:17:36.366775 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366763    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:36.366833 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.366782    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 02:17:36.369413 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.369397    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 02:17:36.371922 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.371909    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 02:17:36.371973 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.371939    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 02:17:36.373287 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373274    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373290    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373296    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373301    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373312    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373320    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373335    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373342    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 02:17:36.373349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373348    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 02:17:36.373555 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373354    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 02:17:36.373555 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373362    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 02:17:36.373555 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.373367    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 02:17:36.374171 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.374159    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 02:17:36.374171 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.374171    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 02:17:36.376711 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.376696    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 02:17:36.376811 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.376787    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 02:17:36.376881 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.376862    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 02:17:36.377728 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.377717    2579 server.go:1268] "Started kubelet"
Sep 05 02:17:36.377823 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.377777    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 02:17:36.377881 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.377838    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 02:17:36.377932 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.377893    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 02:17:36.378443 ip-10-0-138-91 systemd[1]: Started Kubernetes Kubelet.
Sep 05 02:17:36.379636 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.379616    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 02:17:36.380191 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.380177    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 02:17:36.384945 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.383792    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-91.ec2.internal.18d24b2c77a8a98c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-91.ec2.internal,UID:ip-10-0-138-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-91.ec2.internal,},FirstTimestamp:2026-09-05 02:17:36.377694604 +0000 UTC m=+0.349998472,LastTimestamp:2026-09-05 02:17:36.377694604 +0000 UTC m=+0.349998472,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-91.ec2.internal,}"
Sep 05 02:17:36.385556 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.385531    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 02:17:36.388518 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.388504    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 02:17:36.388599 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.388521    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:36.388653 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.388634    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:36.389345 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389328    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 02:17:36.389445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389380    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 02:17:36.389445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389328    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 02:17:36.389445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389329    2579 factory.go:55] Registering systemd factory
Sep 05 02:17:36.389445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389420    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 05 02:17:36.389611 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389497    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 02:17:36.389611 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389507    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 02:17:36.389742 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389729    2579 factory.go:153] Registering CRI-O factory
Sep 05 02:17:36.389792 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389746    2579 factory.go:223] Registration of the crio container factory successfully
Sep 05 02:17:36.389843 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389810    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 02:17:36.389843 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389839    2579 factory.go:103] Registering Raw factory
Sep 05 02:17:36.389928 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.389853    2579 manager.go:1196] Started watching for new ooms in manager
Sep 05 02:17:36.390359 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.390344    2579 manager.go:349] Starting recovery of all containers
Sep 05 02:17:36.398575 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.398530    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 02:17:36.402218 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.402187    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 02:17:36.402356 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.402332    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 02:17:36.403522 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.403470    2579 manager.go:354] Recovery completed
Sep 05 02:17:36.404502 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.404368    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s5v88"
Sep 05 02:17:36.404861 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.404841    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 05 02:17:36.407950 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.407938    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.409270 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.409254    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.409332 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.409282    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.409332 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.409291    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.409877 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.409863    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.409952 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.409863    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 02:17:36.409952 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.409905    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 02:17:36.409952 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.409923    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 02:17:36.410475 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.410459    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.410526 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.410486    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.410526 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.410496    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.411028 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.411015    2579 policy_none.go:49] "None policy: Start"
Sep 05 02:17:36.411028 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.411030    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 02:17:36.411116 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.411039    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 02:17:36.411577 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.411561    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s5v88"
Sep 05 02:17:36.412686 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.412676    2579 policy_none.go:47] "Start"
Sep 05 02:17:36.441220 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.441207    2579 manager.go:337] "Starting Device Plugin manager"
Sep 05 02:17:36.441304 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.441238    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 02:17:36.441304 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.441248    2579 server.go:87] "Starting device plugin registration server"
Sep 05 02:17:36.441519 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.441505    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 02:17:36.441563 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.441522    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 02:17:36.443331 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.441829    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 02:17:36.443331 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.441902    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 02:17:36.443331 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.441915    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 02:17:36.443331 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.442265    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 02:17:36.443331 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.442311    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:36.541740 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.541717    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.542479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.542455    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.542562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.542492    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.542562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.542508    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.542562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.542535    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.544353 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.544330    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 02:17:36.544445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.544372    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 02:17:36.544445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.544390    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 02:17:36.544445 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.544420    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 02:17:36.546327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.546313    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:36.551964 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.551908    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.551964 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.551926    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-91.ec2.internal\": node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:36.600542 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.600527    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:36.644770 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.644751    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal"]
Sep 05 02:17:36.644844 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.644815    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.645696 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.645658    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.645696 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.645694    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.645813 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.645709    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.646894 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.646882    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.647006 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.646991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.647045 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.647016    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.648643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.648628    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.648712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.648633    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.648712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.648678    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.648712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.648692    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.648712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.648694    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.648712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.648708    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.649966 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.649950    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.650029 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.649976    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 02:17:36.654390 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.654374    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 02:17:36.654468 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.654404    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 02:17:36.654468 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.654415    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeHasSufficientPID"
Sep 05 02:17:36.668172 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.668156    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-91.ec2.internal\" not found" node="ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.673752 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.673735    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-91.ec2.internal\" not found" node="ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.691881 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.691863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41169acdb4559aaf703139d914086eec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal\" (UID: \"41169acdb4559aaf703139d914086eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.691978 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.691893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41169acdb4559aaf703139d914086eec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal\" (UID: \"41169acdb4559aaf703139d914086eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.691978 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.691915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8af2bf76f7435a43939d2bafa383cafd-config\") pod \"kube-apiserver-proxy-ip-10-0-138-91.ec2.internal\" (UID: \"8af2bf76f7435a43939d2bafa383cafd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.701039 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.701025    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:36.792269 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.792246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41169acdb4559aaf703139d914086eec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal\" (UID: \"41169acdb4559aaf703139d914086eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.792364 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.792287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41169acdb4559aaf703139d914086eec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal\" (UID: \"41169acdb4559aaf703139d914086eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.792364 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.792313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8af2bf76f7435a43939d2bafa383cafd-config\") pod \"kube-apiserver-proxy-ip-10-0-138-91.ec2.internal\" (UID: \"8af2bf76f7435a43939d2bafa383cafd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.792364 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.792248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/41169acdb4559aaf703139d914086eec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal\" (UID: \"41169acdb4559aaf703139d914086eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.792364 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.792347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8af2bf76f7435a43939d2bafa383cafd-config\") pod \"kube-apiserver-proxy-ip-10-0-138-91.ec2.internal\" (UID: \"8af2bf76f7435a43939d2bafa383cafd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.792497 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.792368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41169acdb4559aaf703139d914086eec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal\" (UID: \"41169acdb4559aaf703139d914086eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.801363 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.801337    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:36.902203 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:36.902151    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:36.970352 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.970323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:36.977327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:36.977307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:37.003081 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.003059    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.103640 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.103613    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.204317 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.204288    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.304459 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.304432    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.322335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.322301    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 02:17:37.322480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.322460    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 02:17:37.405093 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.405068    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.415627 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.415568    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 02:12:37 +0000 UTC" deadline="2028-03-19 01:17:48.703143032 +0000 UTC"
Sep 05 02:17:37.415627 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.415602    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13463h0m11.287544246s"
Sep 05 02:17:37.505814 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.505783    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.606613 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.606592    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.703169 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:37.703012    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af2bf76f7435a43939d2bafa383cafd.slice/crio-d804a7bef4cce903bf70209f666bf9c4d247632fb55e83e341c44d48e21c5a94 WatchSource:0}: Error finding container d804a7bef4cce903bf70209f666bf9c4d247632fb55e83e341c44d48e21c5a94: Status 404 returned error can't find the container with id d804a7bef4cce903bf70209f666bf9c4d247632fb55e83e341c44d48e21c5a94
Sep 05 02:17:37.706741 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.706723    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.707342 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.707324    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:17:37.807353 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.807328    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.907446 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:37.907422    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-91.ec2.internal\" not found"
Sep 05 02:17:37.907547 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.907456    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:37.975170 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.975103    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:37.975484 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.975465    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:37.989116 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:37.989093    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:38.003460 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.003444    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:38.004518 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.004503    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal"
Sep 05 02:17:38.016648 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.016632    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 02:17:38.147596 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.147570    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 02:17:38.370183 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.370116    2579 apiserver.go:52] "Watching apiserver"
Sep 05 02:17:38.376120 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.376094    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 02:17:38.376643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.376613    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-7jm6h","openshift-multus/network-metrics-daemon-xk94l","openshift-network-diagnostics/network-check-target-svblx","kube-system/global-pull-secret-syncer-c75c8","kube-system/konnectivity-agent-tqks4","openshift-cluster-node-tuning-operator/tuned-6md2v","openshift-image-registry/node-ca-2mzt5","openshift-multus/multus-hk2dc","openshift-network-operator/iptables-alerter-s4qjf","openshift-ovn-kubernetes/ovnkube-node-hnk6d","kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85","openshift-dns/node-resolver-6sbhk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal"]
Sep 05 02:17:38.378090 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.378066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.379354 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.379329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.380383 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.380357    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.380504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.380436    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hppp9\""
Sep 05 02:17:38.380568 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.380507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:38.380568 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.380363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.380568 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.380560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 02:17:38.380715 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.380571    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:38.381503 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.381481    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.382022 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.381884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.382022 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.381903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7z7q2\""
Sep 05 02:17:38.382022 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.381963    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.382022 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.381980    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:38.383219 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.383195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.384599 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.384576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.385679 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.385634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 02:17:38.385679 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.385656    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 02:17:38.385841 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.385751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jqjw2\""
Sep 05 02:17:38.386075 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.386057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.387034 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.386769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.387034 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.386809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.387034 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.386770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2lmgp\""
Sep 05 02:17:38.387924 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.387903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.388478 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.388447    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.388570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.388501    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.388570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.388526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 02:17:38.388666 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.388530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vrp77\""
Sep 05 02:17:38.389604 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.388762    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 02:17:38.389604 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.388930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:38.389604 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.388997    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:38.390323 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.390290    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 02:17:38.390323 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.390312    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.391203 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.390616    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5gjhl\""
Sep 05 02:17:38.391203 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.390705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.391203 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.391008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 02:17:38.391203 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.391025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 02:17:38.391410 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.391358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.392476 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.392426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.393805 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.393787    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.393921 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.393896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.394026 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.394001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.394148 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.394097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 02:17:38.394148 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.394118    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j4j6n\""
Sep 05 02:17:38.395778 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.395744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 02:17:38.395935 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.395912    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dwnwz\""
Sep 05 02:17:38.398603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.396923    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 02:17:38.398603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.397142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 02:17:38.398603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.397822    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 02:17:38.398603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.398316    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 02:17:38.398603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.398506    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 02:17:38.399024 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.398682    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xs6k7\""
Sep 05 02:17:38.399024 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.398776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 02:17:38.400122 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httsn\" (UniqueName: \"kubernetes.io/projected/4662252d-cd26-4d5c-ad62-0c815a71e462-kube-api-access-httsn\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:38.400236 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8f03596-1d37-470e-86e3-0b2ae5138bdf-serviceca\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.400293 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-env-overrides\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.400360 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxz5x\" (UniqueName: \"kubernetes.io/projected/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-kube-api-access-dxz5x\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.400415 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00d08ef6-5502-4c36-9da3-e33abed6ba56-konnectivity-ca\") pod \"konnectivity-agent-tqks4\" (UID: \"00d08ef6-5502-4c36-9da3-e33abed6ba56\") " pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.400415 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-systemd\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.400524 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysctl-d\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.400524 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6jk\" (UniqueName: \"kubernetes.io/projected/d8f03596-1d37-470e-86e3-0b2ae5138bdf-kube-api-access-fq6jk\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.400524 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-run-netns\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.400644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-ovnkube-config\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.400644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:38.400644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.400644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.400836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-system-cni-dir\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.400836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.400836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-slash\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.400836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-lib-modules\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.400836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.401081 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-cni-netd\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.401081 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-run\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.401081 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5150c3b3-98c7-4bd7-b469-950ca5c40029-tmp\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.401081 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8f03596-1d37-470e-86e3-0b2ae5138bdf-host\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.401081 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.400979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysctl-conf\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.401081 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7nf\" (UniqueName: \"kubernetes.io/projected/5150c3b3-98c7-4bd7-b469-950ca5c40029-kube-api-access-xn7nf\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.401081 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-kubelet-config\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-node-log\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-hosts-file\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysconfig\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-systemd\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-kubelet\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-cni-bin\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlnb\" (UniqueName: \"kubernetes.io/projected/88034fda-9465-4196-bd5e-9ca3907b9bbb-kube-api-access-qnlnb\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-tmp-dir\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cnibin\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqf72\" (UniqueName: \"kubernetes.io/projected/a4d66a85-c9db-43a2-8465-62b8a630df97-kube-api-access-nqf72\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.401479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-modprobe-d\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-var-lib-kubelet\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-host\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-tuned\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88034fda-9465-4196-bd5e-9ca3907b9bbb-iptables-alerter-script\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-dbus\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-etc-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-log-socket\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-kubernetes\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-systemd-units\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-var-lib-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-ovn\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4d66a85-c9db-43a2-8465-62b8a630df97-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-ovnkube-script-lib\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00d08ef6-5502-4c36-9da3-e33abed6ba56-agent-certs\") pod \"konnectivity-agent-tqks4\" (UID: \"00d08ef6-5502-4c36-9da3-e33abed6ba56\") " pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.401965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-sys\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.402117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.402004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88034fda-9465-4196-bd5e-9ca3907b9bbb-host-slash\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.402876 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.402037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:38.402876 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.402061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.402876 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.402098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm29m\" (UniqueName: \"kubernetes.io/projected/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-kube-api-access-xm29m\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.402876 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.402165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.402876 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.402231    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 02:17:38.422512 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.422480    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q5pcx"
Sep 05 02:17:38.430948 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.430927    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q5pcx"
Sep 05 02:17:38.490709 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.490687    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 02:17:38.502359 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysconfig\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.502472 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-systemd\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.502472 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.502472 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-kubelet\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.502472 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-cni-bin\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-netns\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-registration-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlnb\" (UniqueName: \"kubernetes.io/projected/88034fda-9465-4196-bd5e-9ca3907b9bbb-kube-api-access-qnlnb\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-tmp-dir\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-cni-bin\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cnibin\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqf72\" (UniqueName: \"kubernetes.io/projected/a4d66a85-c9db-43a2-8465-62b8a630df97-kube-api-access-nqf72\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-modprobe-d\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysconfig\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-var-lib-kubelet\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-modprobe-d\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-host\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-tuned\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-host\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88034fda-9465-4196-bd5e-9ca3907b9bbb-iptables-alerter-script\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.504063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-kubelet\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-dbus\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-cni-bin\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-etc-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.502985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-log-socket\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-var-lib-kubelet\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-kubernetes\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-cnibin\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-systemd\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-cni-multus\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-conf-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-daemon-config\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4psfs\" (UniqueName: \"kubernetes.io/projected/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-kube-api-access-4psfs\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-systemd-units\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-var-lib-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-ovn\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4d66a85-c9db-43a2-8465-62b8a630df97-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505049 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-ovnkube-script-lib\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00d08ef6-5502-4c36-9da3-e33abed6ba56-agent-certs\") pod \"konnectivity-agent-tqks4\" (UID: \"00d08ef6-5502-4c36-9da3-e33abed6ba56\") " pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-kubernetes\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-sys-fs\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-sys\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88034fda-9465-4196-bd5e-9ca3907b9bbb-host-slash\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm29m\" (UniqueName: \"kubernetes.io/projected/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-kube-api-access-xm29m\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93b93690-899f-4fcc-a6da-cb1fe002b30b-cni-binary-copy\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-etc-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-log-socket\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httsn\" (UniqueName: \"kubernetes.io/projected/4662252d-cd26-4d5c-ad62-0c815a71e462-kube-api-access-httsn\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8f03596-1d37-470e-86e3-0b2ae5138bdf-serviceca\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-dbus\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.505985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503675    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cnibin\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.503774    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.503880    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret podName:4183dcc7-2263-45e0-97e8-ad25a6c7ad9a nodeName:}" failed. No retries permitted until 2026-09-05 02:17:39.003836884 +0000 UTC m=+2.976140758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret") pod "global-pull-secret-syncer-c75c8" (UID: "4183dcc7-2263-45e0-97e8-ad25a6c7ad9a") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-ovnkube-script-lib\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-var-lib-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-ovn\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88034fda-9465-4196-bd5e-9ca3907b9bbb-host-slash\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-env-overrides\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxz5x\" (UniqueName: \"kubernetes.io/projected/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-kube-api-access-dxz5x\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00d08ef6-5502-4c36-9da3-e33abed6ba56-konnectivity-ca\") pod \"konnectivity-agent-tqks4\" (UID: \"00d08ef6-5502-4c36-9da3-e33abed6ba56\") " pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-cni-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-device-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-systemd\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-tmp-dir\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.506885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-socket-dir-parent\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-systemd-units\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-k8s-cni-cncf-io\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysctl-d\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6jk\" (UniqueName: \"kubernetes.io/projected/d8f03596-1d37-470e-86e3-0b2ae5138bdf-kube-api-access-fq6jk\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-run-netns\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-ovnkube-config\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.504925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-kubelet\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-hostroot\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8f03596-1d37-470e-86e3-0b2ae5138bdf-serviceca\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-etc-kubernetes\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmw8\" (UniqueName: \"kubernetes.io/projected/93b93690-899f-4fcc-a6da-cb1fe002b30b-kube-api-access-2xmw8\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-system-cni-dir\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.507689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-slash\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-lib-modules\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00d08ef6-5502-4c36-9da3-e33abed6ba56-konnectivity-ca\") pod \"konnectivity-agent-tqks4\" (UID: \"00d08ef6-5502-4c36-9da3-e33abed6ba56\") " pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-cni-netd\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-socket-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-run\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5150c3b3-98c7-4bd7-b469-950ca5c40029-tmp\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8f03596-1d37-470e-86e3-0b2ae5138bdf-host\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-system-cni-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-multus-certs\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-env-overrides\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysctl-conf\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7nf\" (UniqueName: \"kubernetes.io/projected/5150c3b3-98c7-4bd7-b469-950ca5c40029-kube-api-access-xn7nf\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-systemd\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.505755    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:38.508651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-system-cni-dir\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8f03596-1d37-470e-86e3-0b2ae5138bdf-host\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.505842    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:39.00582725 +0000 UTC m=+2.978131123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-lib-modules\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysctl-conf\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-cni-netd\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-run-netns\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-kubelet-config\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-kubelet-config\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-sysctl-d\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-node-log\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-run\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-hosts-file\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-host-slash\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.509626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.505771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-run-openvswitch\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.510562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a4d66a85-c9db-43a2-8465-62b8a630df97-ovnkube-config\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.510562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a4d66a85-c9db-43a2-8465-62b8a630df97-node-log\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.510562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.506815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-hosts-file\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.510562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.503699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5150c3b3-98c7-4bd7-b469-950ca5c40029-sys\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.510562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.507404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88034fda-9465-4196-bd5e-9ca3907b9bbb-iptables-alerter-script\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.510893 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.510870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5150c3b3-98c7-4bd7-b469-950ca5c40029-etc-tuned\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.511384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.511360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00d08ef6-5502-4c36-9da3-e33abed6ba56-agent-certs\") pod \"konnectivity-agent-tqks4\" (UID: \"00d08ef6-5502-4c36-9da3-e33abed6ba56\") " pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.512108 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.512086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5150c3b3-98c7-4bd7-b469-950ca5c40029-tmp\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.512219 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.512203    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:38.512293 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.512226    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:38.512293 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.512239    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7z2j2 for pod openshift-network-diagnostics/network-check-target-svblx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:38.512293 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:38.512292    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2 podName:59e79702-89d7-49a0-bd00-8754b43eb10d nodeName:}" failed. No retries permitted until 2026-09-05 02:17:39.012275192 +0000 UTC m=+2.984579062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7z2j2" (UniqueName: "kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2") pod "network-check-target-svblx" (UID: "59e79702-89d7-49a0-bd00-8754b43eb10d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:38.512940 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.512918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4d66a85-c9db-43a2-8465-62b8a630df97-ovn-node-metrics-cert\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.517053 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.516981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlnb\" (UniqueName: \"kubernetes.io/projected/88034fda-9465-4196-bd5e-9ca3907b9bbb-kube-api-access-qnlnb\") pod \"iptables-alerter-s4qjf\" (UID: \"88034fda-9465-4196-bd5e-9ca3907b9bbb\") " pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.517157 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.517120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxz5x\" (UniqueName: \"kubernetes.io/projected/a7eed417-1f97-47b2-b2e2-6ec960bbfda3-kube-api-access-dxz5x\") pod \"node-resolver-6sbhk\" (UID: \"a7eed417-1f97-47b2-b2e2-6ec960bbfda3\") " pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.518465 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.518420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7nf\" (UniqueName: \"kubernetes.io/projected/5150c3b3-98c7-4bd7-b469-950ca5c40029-kube-api-access-xn7nf\") pod \"tuned-6md2v\" (UID: \"5150c3b3-98c7-4bd7-b469-950ca5c40029\") " pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.519497 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.519459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqf72\" (UniqueName: \"kubernetes.io/projected/a4d66a85-c9db-43a2-8465-62b8a630df97-kube-api-access-nqf72\") pod \"ovnkube-node-hnk6d\" (UID: \"a4d66a85-c9db-43a2-8465-62b8a630df97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.520187 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.520144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6jk\" (UniqueName: \"kubernetes.io/projected/d8f03596-1d37-470e-86e3-0b2ae5138bdf-kube-api-access-fq6jk\") pod \"node-ca-2mzt5\" (UID: \"d8f03596-1d37-470e-86e3-0b2ae5138bdf\") " pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.520563 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.520528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm29m\" (UniqueName: \"kubernetes.io/projected/cb5c424f-d9a6-4d3c-a4b1-c26857227b44-kube-api-access-xm29m\") pod \"multus-additional-cni-plugins-7jm6h\" (UID: \"cb5c424f-d9a6-4d3c-a4b1-c26857227b44\") " pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.521288 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.521265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-httsn\" (UniqueName: \"kubernetes.io/projected/4662252d-cd26-4d5c-ad62-0c815a71e462-kube-api-access-httsn\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:38.552320 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.552270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal" event={"ID":"8af2bf76f7435a43939d2bafa383cafd","Type":"ContainerStarted","Data":"d804a7bef4cce903bf70209f666bf9c4d247632fb55e83e341c44d48e21c5a94"}
Sep 05 02:17:38.554062 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.554035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal" event={"ID":"41169acdb4559aaf703139d914086eec","Type":"ContainerStarted","Data":"85f04b1c9f43b945be0899c39bcb82bdbe5b3895e807d9a46a479f62d983c77b"}
Sep 05 02:17:38.607248 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-cni-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607367 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-device-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607367 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-socket-dir-parent\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607367 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-k8s-cni-cncf-io\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607367 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-kubelet\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-hostroot\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-etc-kubernetes\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-cni-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmw8\" (UniqueName: \"kubernetes.io/projected/93b93690-899f-4fcc-a6da-cb1fe002b30b-kube-api-access-2xmw8\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-socket-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607573 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-system-cni-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-multus-certs\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-netns\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-registration-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-cni-bin\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-cnibin\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-cni-multus\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-device-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-conf-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-kubelet\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-conf-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-daemon-config\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-etc-kubernetes\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-socket-dir-parent\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4psfs\" (UniqueName: \"kubernetes.io/projected/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-kube-api-access-4psfs\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-hostroot\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-sys-fs\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.607985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-k8s-cni-cncf-io\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93b93690-899f-4fcc-a6da-cb1fe002b30b-cni-binary-copy\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.607943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-socket-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-netns\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-registration-dir\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-cni-bin\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-cnibin\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-var-lib-cni-multus\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-system-cni-dir\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-sys-fs\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/93b93690-899f-4fcc-a6da-cb1fe002b30b-host-run-multus-certs\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.608900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.608446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/93b93690-899f-4fcc-a6da-cb1fe002b30b-multus-daemon-config\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.609501 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.609216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93b93690-899f-4fcc-a6da-cb1fe002b30b-cni-binary-copy\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.623644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.623589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmw8\" (UniqueName: \"kubernetes.io/projected/93b93690-899f-4fcc-a6da-cb1fe002b30b-kube-api-access-2xmw8\") pod \"multus-hk2dc\" (UID: \"93b93690-899f-4fcc-a6da-cb1fe002b30b\") " pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.626256 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.626223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4psfs\" (UniqueName: \"kubernetes.io/projected/01c0cedd-46c8-419d-b0fc-027b1d3d0b12-kube-api-access-4psfs\") pod \"aws-ebs-csi-driver-node-7jc85\" (UID: \"01c0cedd-46c8-419d-b0fc-027b1d3d0b12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.692466 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.692446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2mzt5"
Sep 05 02:17:38.702547 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.702522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6sbhk"
Sep 05 02:17:38.705058 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.705029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f03596_1d37_470e_86e3_0b2ae5138bdf.slice/crio-51ecbe66b31e6180b42ae9980532649aad8d901f32bdcb98c0dd0c51059aa932 WatchSource:0}: Error finding container 51ecbe66b31e6180b42ae9980532649aad8d901f32bdcb98c0dd0c51059aa932: Status 404 returned error can't find the container with id 51ecbe66b31e6180b42ae9980532649aad8d901f32bdcb98c0dd0c51059aa932
Sep 05 02:17:38.714346 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.714324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:17:38.716381 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.716347    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eed417_1f97_47b2_b2e2_6ec960bbfda3.slice/crio-502812d4b9d02a5bb2b54522ef113b2a166be0049dccb01032628854c473b1f9 WatchSource:0}: Error finding container 502812d4b9d02a5bb2b54522ef113b2a166be0049dccb01032628854c473b1f9: Status 404 returned error can't find the container with id 502812d4b9d02a5bb2b54522ef113b2a166be0049dccb01032628854c473b1f9
Sep 05 02:17:38.719507 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.719278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6md2v"
Sep 05 02:17:38.725255 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.725219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d08ef6_5502_4c36_9da3_e33abed6ba56.slice/crio-54f4189ea77ea1290ec611e1b23655564ee1c6d1aa59defb5a9c8286b7c494e3 WatchSource:0}: Error finding container 54f4189ea77ea1290ec611e1b23655564ee1c6d1aa59defb5a9c8286b7c494e3: Status 404 returned error can't find the container with id 54f4189ea77ea1290ec611e1b23655564ee1c6d1aa59defb5a9c8286b7c494e3
Sep 05 02:17:38.731099 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.731078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4qjf"
Sep 05 02:17:38.732234 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.732214    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5150c3b3_98c7_4bd7_b469_950ca5c40029.slice/crio-124a261fa39274875c099f5413776eab99b052fa9a109787f06578073e023a06 WatchSource:0}: Error finding container 124a261fa39274875c099f5413776eab99b052fa9a109787f06578073e023a06: Status 404 returned error can't find the container with id 124a261fa39274875c099f5413776eab99b052fa9a109787f06578073e023a06
Sep 05 02:17:38.741195 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.741172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jm6h"
Sep 05 02:17:38.752350 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.752332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:17:38.753399 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.753369    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88034fda_9465_4196_bd5e_9ca3907b9bbb.slice/crio-75595afc305625c3242fe41696da49ec2f8b77e766fd5fd0e05ce80b3f0569a0 WatchSource:0}: Error finding container 75595afc305625c3242fe41696da49ec2f8b77e766fd5fd0e05ce80b3f0569a0: Status 404 returned error can't find the container with id 75595afc305625c3242fe41696da49ec2f8b77e766fd5fd0e05ce80b3f0569a0
Sep 05 02:17:38.759383 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.759361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk2dc"
Sep 05 02:17:38.759839 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.759813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5c424f_d9a6_4d3c_a4b1_c26857227b44.slice/crio-4db395dfcdb9fe805d4c51a66dbe3bcf87b7df7523487d2b15de02ebf1cbb7fc WatchSource:0}: Error finding container 4db395dfcdb9fe805d4c51a66dbe3bcf87b7df7523487d2b15de02ebf1cbb7fc: Status 404 returned error can't find the container with id 4db395dfcdb9fe805d4c51a66dbe3bcf87b7df7523487d2b15de02ebf1cbb7fc
Sep 05 02:17:38.765903 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:38.765881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85"
Sep 05 02:17:38.766384 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.766361    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d66a85_c9db_43a2_8465_62b8a630df97.slice/crio-a4a437fee960786ccc8e18e5a2543795850e9c0202f2b395d8e2ae09c68cb75d WatchSource:0}: Error finding container a4a437fee960786ccc8e18e5a2543795850e9c0202f2b395d8e2ae09c68cb75d: Status 404 returned error can't find the container with id a4a437fee960786ccc8e18e5a2543795850e9c0202f2b395d8e2ae09c68cb75d
Sep 05 02:17:38.772719 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.772695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b93690_899f_4fcc_a6da_cb1fe002b30b.slice/crio-f13c21f5125de5f287b1e9ffaa14587ea75e0c0d975f6ca354188fb3c481308a WatchSource:0}: Error finding container f13c21f5125de5f287b1e9ffaa14587ea75e0c0d975f6ca354188fb3c481308a: Status 404 returned error can't find the container with id f13c21f5125de5f287b1e9ffaa14587ea75e0c0d975f6ca354188fb3c481308a
Sep 05 02:17:38.779746 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:17:38.779675    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c0cedd_46c8_419d_b0fc_027b1d3d0b12.slice/crio-79ca12f4baad0d532c7b712f156228a5f839167a0ba6abaaeee4b81758089483 WatchSource:0}: Error finding container 79ca12f4baad0d532c7b712f156228a5f839167a0ba6abaaeee4b81758089483: Status 404 returned error can't find the container with id 79ca12f4baad0d532c7b712f156228a5f839167a0ba6abaaeee4b81758089483
Sep 05 02:17:39.013777 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.013703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:39.013777 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.013740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:39.013986 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.013865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:39.013986 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.013881    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:39.013986 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.013882    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:39.013986 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.013953    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:40.013932553 +0000 UTC m=+3.986236412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:39.013986 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.013980    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret podName:4183dcc7-2263-45e0-97e8-ad25a6c7ad9a nodeName:}" failed. No retries permitted until 2026-09-05 02:17:40.013962386 +0000 UTC m=+3.986266261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret") pod "global-pull-secret-syncer-c75c8" (UID: "4183dcc7-2263-45e0-97e8-ad25a6c7ad9a") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:39.013986 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.013885    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:39.014243 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.014010    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:39.014243 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.014022    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7z2j2 for pod openshift-network-diagnostics/network-check-target-svblx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:39.014243 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.014061    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2 podName:59e79702-89d7-49a0-bd00-8754b43eb10d nodeName:}" failed. No retries permitted until 2026-09-05 02:17:40.014050705 +0000 UTC m=+3.986354561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z2j2" (UniqueName: "kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2") pod "network-check-target-svblx" (UID: "59e79702-89d7-49a0-bd00-8754b43eb10d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:39.434666 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.434575    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:38 +0000 UTC" deadline="2028-02-28 14:13:31.650939937 +0000 UTC"
Sep 05 02:17:39.434666 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.434620    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12995h55m52.216326248s"
Sep 05 02:17:39.544952 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.544918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:39.545125 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:39.545063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:39.559845 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.559750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85" event={"ID":"01c0cedd-46c8-419d-b0fc-027b1d3d0b12","Type":"ContainerStarted","Data":"79ca12f4baad0d532c7b712f156228a5f839167a0ba6abaaeee4b81758089483"}
Sep 05 02:17:39.561976 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.561905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk2dc" event={"ID":"93b93690-899f-4fcc-a6da-cb1fe002b30b","Type":"ContainerStarted","Data":"f13c21f5125de5f287b1e9ffaa14587ea75e0c0d975f6ca354188fb3c481308a"}
Sep 05 02:17:39.563579 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.563509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6md2v" event={"ID":"5150c3b3-98c7-4bd7-b469-950ca5c40029","Type":"ContainerStarted","Data":"124a261fa39274875c099f5413776eab99b052fa9a109787f06578073e023a06"}
Sep 05 02:17:39.565806 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.565750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tqks4" event={"ID":"00d08ef6-5502-4c36-9da3-e33abed6ba56","Type":"ContainerStarted","Data":"54f4189ea77ea1290ec611e1b23655564ee1c6d1aa59defb5a9c8286b7c494e3"}
Sep 05 02:17:39.567565 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.567477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6sbhk" event={"ID":"a7eed417-1f97-47b2-b2e2-6ec960bbfda3","Type":"ContainerStarted","Data":"502812d4b9d02a5bb2b54522ef113b2a166be0049dccb01032628854c473b1f9"}
Sep 05 02:17:39.570223 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.570167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2mzt5" event={"ID":"d8f03596-1d37-470e-86e3-0b2ae5138bdf","Type":"ContainerStarted","Data":"51ecbe66b31e6180b42ae9980532649aad8d901f32bdcb98c0dd0c51059aa932"}
Sep 05 02:17:39.572384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.572357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"a4a437fee960786ccc8e18e5a2543795850e9c0202f2b395d8e2ae09c68cb75d"}
Sep 05 02:17:39.574082 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.574058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerStarted","Data":"4db395dfcdb9fe805d4c51a66dbe3bcf87b7df7523487d2b15de02ebf1cbb7fc"}
Sep 05 02:17:39.575819 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:39.575794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4qjf" event={"ID":"88034fda-9465-4196-bd5e-9ca3907b9bbb","Type":"ContainerStarted","Data":"75595afc305625c3242fe41696da49ec2f8b77e766fd5fd0e05ce80b3f0569a0"}
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:40.021889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:40.021940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:40.021984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022150    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022168    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022181    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7z2j2 for pod openshift-network-diagnostics/network-check-target-svblx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022240    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2 podName:59e79702-89d7-49a0-bd00-8754b43eb10d nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.022220095 +0000 UTC m=+5.994523974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z2j2" (UniqueName: "kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2") pod "network-check-target-svblx" (UID: "59e79702-89d7-49a0-bd00-8754b43eb10d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022672    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022726    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.022709846 +0000 UTC m=+5.995013718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022803    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:40.022901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.022838    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret podName:4183dcc7-2263-45e0-97e8-ad25a6c7ad9a nodeName:}" failed. No retries permitted until 2026-09-05 02:17:42.02282709 +0000 UTC m=+5.995130961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret") pod "global-pull-secret-syncer-c75c8" (UID: "4183dcc7-2263-45e0-97e8-ad25a6c7ad9a") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:40.435173 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:40.434962    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 02:12:38 +0000 UTC" deadline="2028-04-24 07:02:03.204557464 +0000 UTC"
Sep 05 02:17:40.435173 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:40.435021    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14332h44m22.7695429s"
Sep 05 02:17:40.547688 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:40.547655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:40.547875 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.547809    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:40.548254 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:40.548233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:40.548340 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:40.548326    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:41.545754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:41.545228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:41.545754 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:41.545332    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:41.602435 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:41.602397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal" event={"ID":"8af2bf76f7435a43939d2bafa383cafd","Type":"ContainerStarted","Data":"bc025be380448195e6ea180a29b708c7d2911dfc4464c33b7570667256642899"}
Sep 05 02:17:42.041923 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:42.041885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:42.042072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:42.041940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:42.042072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:42.041982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:42.042198 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042134    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:42.042198 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042156    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:42.042198 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042170    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7z2j2 for pod openshift-network-diagnostics/network-check-target-svblx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:42.042348 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042227    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2 podName:59e79702-89d7-49a0-bd00-8754b43eb10d nodeName:}" failed. No retries permitted until 2026-09-05 02:17:46.042207605 +0000 UTC m=+10.014511476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z2j2" (UniqueName: "kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2") pod "network-check-target-svblx" (UID: "59e79702-89d7-49a0-bd00-8754b43eb10d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:42.042658 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042634    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.042771 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042700    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:46.042683085 +0000 UTC m=+10.014986959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:42.042847 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042780    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.042847 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.042821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret podName:4183dcc7-2263-45e0-97e8-ad25a6c7ad9a nodeName:}" failed. No retries permitted until 2026-09-05 02:17:46.042808018 +0000 UTC m=+10.015111892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret") pod "global-pull-secret-syncer-c75c8" (UID: "4183dcc7-2263-45e0-97e8-ad25a6c7ad9a") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:42.547163 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:42.547132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:42.547669 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.547264    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:42.547756 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:42.547672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:42.547816 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:42.547783    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:42.607749 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:42.607636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal" event={"ID":"41169acdb4559aaf703139d914086eec","Type":"ContainerStarted","Data":"1b311949a794679a5fade56d5fad31976d2f6813a6a9d3427d71fdf1f813a519"}
Sep 05 02:17:42.625521 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:42.625206    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-91.ec2.internal" podStartSLOduration=4.62518497 podStartE2EDuration="4.62518497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:41.62832397 +0000 UTC m=+5.600627850" watchObservedRunningTime="2026-09-05 02:17:42.62518497 +0000 UTC m=+6.597488853"
Sep 05 02:17:43.545603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:43.545555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:43.545820 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:43.545723    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:44.546419 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:44.546105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:44.546905 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:44.546122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:44.546905 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:44.546493    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:44.546905 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:44.546578    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:44.615842 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:44.615808    2579 generic.go:356] "Generic (PLEG): container finished" podID="41169acdb4559aaf703139d914086eec" containerID="1b311949a794679a5fade56d5fad31976d2f6813a6a9d3427d71fdf1f813a519" exitCode=0
Sep 05 02:17:44.616012 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:44.615870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal" event={"ID":"41169acdb4559aaf703139d914086eec","Type":"ContainerDied","Data":"1b311949a794679a5fade56d5fad31976d2f6813a6a9d3427d71fdf1f813a519"}
Sep 05 02:17:45.545402 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:45.545364    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:45.545553 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:45.545521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:46.080489 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:46.080454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:46.080531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:46.080555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080617    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080643    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080655    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7z2j2 for pod openshift-network-diagnostics/network-check-target-svblx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080680    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080717    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2 podName:59e79702-89d7-49a0-bd00-8754b43eb10d nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.080699213 +0000 UTC m=+18.053003079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z2j2" (UniqueName: "kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2") pod "network-check-target-svblx" (UID: "59e79702-89d7-49a0-bd00-8754b43eb10d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080736    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret podName:4183dcc7-2263-45e0-97e8-ad25a6c7ad9a nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.080729497 +0000 UTC m=+18.053033351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret") pod "global-pull-secret-syncer-c75c8" (UID: "4183dcc7-2263-45e0-97e8-ad25a6c7ad9a") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080772    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:46.080979 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.080811    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:17:54.080795141 +0000 UTC m=+18.053099009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:46.545556 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:46.545524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:46.545765 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.545648    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:46.546142 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:46.545998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:46.546142 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:46.546097    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:47.545005 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:47.544962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:47.545411 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:47.545102    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:48.545339 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:48.545305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:48.545820 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:48.545420    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:48.545820 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:48.545462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:48.545820 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:48.545527    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:49.544901 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:49.544868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:49.545069 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:49.544992    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:50.544956 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:50.544923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:50.545404 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:50.545043    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:50.545404 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:50.545112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:50.545404 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:50.545225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:51.545380 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:51.545344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:51.545818 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:51.545459    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:52.544756 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:52.544696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:52.544931 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:52.544815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:52.544931 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:52.544813    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:52.544931 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:52.544914    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:53.545334 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:53.545304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:53.545777 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:53.545410    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:54.142417 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:54.142384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:54.142417 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:54.142416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:54.142647 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:54.142440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:54.142647 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142550    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:54.142647 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142587    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:17:54.142647 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142601    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:17:54.142647 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142613    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7z2j2 for pod openshift-network-diagnostics/network-check-target-svblx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:54.142647 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142616    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret podName:4183dcc7-2263-45e0-97e8-ad25a6c7ad9a nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.142597347 +0000 UTC m=+34.114901217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret") pod "global-pull-secret-syncer-c75c8" (UID: "4183dcc7-2263-45e0-97e8-ad25a6c7ad9a") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:17:54.142647 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142551    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:54.142882 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2 podName:59e79702-89d7-49a0-bd00-8754b43eb10d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.14263496 +0000 UTC m=+34.114938817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z2j2" (UniqueName: "kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2") pod "network-check-target-svblx" (UID: "59e79702-89d7-49a0-bd00-8754b43eb10d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:17:54.142882 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.142709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.142693782 +0000 UTC m=+34.114997636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:17:54.545690 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:54.545590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:54.546135 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:54.545590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:54.546135 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.545724    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:54.546135 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:54.545798    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:55.544702 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:55.544675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:55.544861 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:55.544777    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:56.546333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:56.546302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:56.546754 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:56.546413    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:56.546754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:56.546474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:56.546754 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:56.546589    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:57.545346 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.545214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:57.545459 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:57.545435    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:57.640950 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.640921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6md2v" event={"ID":"5150c3b3-98c7-4bd7-b469-950ca5c40029","Type":"ContainerStarted","Data":"8459f04596127d9c09ffb1be8ed2a2adb5508cf6f63f3524d3e523f88ee4a28f"}
Sep 05 02:17:57.642399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.642378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tqks4" event={"ID":"00d08ef6-5502-4c36-9da3-e33abed6ba56","Type":"ContainerStarted","Data":"b1ff7f7fd1aa45cd0c2026da1d3e2994270818fe4d645880f1864c06a011f534"}
Sep 05 02:17:57.644078 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.644050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6sbhk" event={"ID":"a7eed417-1f97-47b2-b2e2-6ec960bbfda3","Type":"ContainerStarted","Data":"9502bc41634c1563f8f082631465d5ce8f9b7af1e4df5658fd9ee1dce3c49c07"}
Sep 05 02:17:57.645527 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.645503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2mzt5" event={"ID":"d8f03596-1d37-470e-86e3-0b2ae5138bdf","Type":"ContainerStarted","Data":"4eb0e01407fd5b8971244b4f74d28f1e3a56c55d1248b1d6b12b8b9b7fd275d8"}
Sep 05 02:17:57.647317 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.647296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal" event={"ID":"41169acdb4559aaf703139d914086eec","Type":"ContainerStarted","Data":"39ebd1700582c4970f34ea93a9eb96c0487b3f645f9ebdc1185510eaeeb8233b"}
Sep 05 02:17:57.649450 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.649424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"11c30522dfe36cf6c0ce2b605c92d1c6f648e9b1d2f57e353995eae54ba82482"}
Sep 05 02:17:57.649538 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.649481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"17a8e166c90ad4e60bc712f42670c95fa5a068aec9ee74a13c3812e137c7d9bd"}
Sep 05 02:17:57.649538 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.649530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"71c2246a92028c3fc4647d02d703cd04fe55b1938cf9a5cd67b901f9bb3d79cc"}
Sep 05 02:17:57.650777 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.650754    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5c424f-d9a6-4d3c-a4b1-c26857227b44" containerID="3fcd6eacc3cf10a5a81ae4f2d9aae6252e626497140b36d2ce3075dc4adc0da0" exitCode=0
Sep 05 02:17:57.650851 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.650830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerDied","Data":"3fcd6eacc3cf10a5a81ae4f2d9aae6252e626497140b36d2ce3075dc4adc0da0"}
Sep 05 02:17:57.652237 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.652223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85" event={"ID":"01c0cedd-46c8-419d-b0fc-027b1d3d0b12","Type":"ContainerStarted","Data":"124cf399f960e2baf0ed44e3bdad0d8b36708426ee3b34e978157e065e6ca1d0"}
Sep 05 02:17:57.653564 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.653545    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk2dc" event={"ID":"93b93690-899f-4fcc-a6da-cb1fe002b30b","Type":"ContainerStarted","Data":"2a5d194c62663cf7024d32b521a35de4ff08ec0a969c1db15b1a84e9b32498b3"}
Sep 05 02:17:57.668399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.668366    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6md2v" podStartSLOduration=2.309462416 podStartE2EDuration="20.668355417s" totalImagesPullingTime="18.358893001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:57.668251468 +0000 UTC m=+21.640555345" watchObservedRunningTime="2026-09-05 02:17:57.668355417 +0000 UTC m=+21.640659292"
Sep 05 02:17:57.732233 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.732194    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2mzt5" podStartSLOduration=2.3519717030000002 podStartE2EDuration="20.732182332s" totalImagesPullingTime="18.380210629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:57.709442438 +0000 UTC m=+21.681746316" watchObservedRunningTime="2026-09-05 02:17:57.732182332 +0000 UTC m=+21.704486206"
Sep 05 02:17:57.764467 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.764434    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tqks4" podStartSLOduration=2.399991799 podStartE2EDuration="20.764408236s" totalImagesPullingTime="18.364416437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:57.732306329 +0000 UTC m=+21.704610186" watchObservedRunningTime="2026-09-05 02:17:57.764408236 +0000 UTC m=+21.736712091"
Sep 05 02:17:57.764615 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.764595    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hk2dc" podStartSLOduration=2.406917818 podStartE2EDuration="20.764588859s" totalImagesPullingTime="18.357671041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:57.764260062 +0000 UTC m=+21.736563941" watchObservedRunningTime="2026-09-05 02:17:57.764588859 +0000 UTC m=+21.736892735"
Sep 05 02:17:57.800849 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.800696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-91.ec2.internal" podStartSLOduration=19.800685326 podStartE2EDuration="19.800685326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:57.800519744 +0000 UTC m=+21.772823620" watchObservedRunningTime="2026-09-05 02:17:57.800685326 +0000 UTC m=+21.772989199"
Sep 05 02:17:57.821991 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:57.821956    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6sbhk" podStartSLOduration=2.708787913 podStartE2EDuration="20.821945285s" totalImagesPullingTime="18.113157372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:57.821715185 +0000 UTC m=+21.794019054" watchObservedRunningTime="2026-09-05 02:17:57.821945285 +0000 UTC m=+21.794249161"
Sep 05 02:17:58.545143 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.545115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:17:58.545353 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.545122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:17:58.545353 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:58.545222    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:17:58.545353 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:58.545312    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:17:58.660091 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.660061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"28aa19ce34de808f1b80b1d2620cfe229159f4c7085b6619f9a94fedcf5ff526"}
Sep 05 02:17:58.660447 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.660130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"29d1db48bc87f651ab53a982fd63b905632a721816e46d852f73ef678ff35197"}
Sep 05 02:17:58.660447 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.660180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"f4e1f1cf3be46b131b69ffaed0d7967c70b06a0e7a17ae746ade0aaedf3900d9"}
Sep 05 02:17:58.661753 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.661727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4qjf" event={"ID":"88034fda-9465-4196-bd5e-9ca3907b9bbb","Type":"ContainerStarted","Data":"0ee0e134fb3ba9ec5bdb349e51393f8c4d3ff86f0ef52f1da4b7a43ab4acf469"}
Sep 05 02:17:58.678223 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.678180    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s4qjf" podStartSLOduration=3.598760187 podStartE2EDuration="21.678166836s" totalImagesPullingTime="18.079406649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:17:58.677806333 +0000 UTC m=+22.650110208" watchObservedRunningTime="2026-09-05 02:17:58.678166836 +0000 UTC m=+22.650470712"
Sep 05 02:17:58.721399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:58.721375    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 02:17:59.456380 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:59.456269    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T02:17:58.721393121Z","UUID":"719727d8-0358-4f39-b60d-238c7a821011","Handler":null,"Name":"","Endpoint":""}
Sep 05 02:17:59.458053 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:59.458030    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 02:17:59.458182 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:59.458065    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 02:17:59.545653 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:59.545621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:17:59.545825 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:17:59.545799    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:17:59.666726 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:17:59.666681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85" event={"ID":"01c0cedd-46c8-419d-b0fc-027b1d3d0b12","Type":"ContainerStarted","Data":"8a82d0c1ca8c0d743869e34dfa2c8279d9b6599e38e479167cc19768308c66b9"}
Sep 05 02:18:00.544797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:00.544770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:00.544797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:00.544794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:00.545084 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:00.544878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:18:00.545084 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:00.544990    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:18:00.673427 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:00.673392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"d3c28cdeb28d83d1af89d2db4dd332a6e2a275c24f4ae663cf62541fbb6042d9"}
Sep 05 02:18:00.675680 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:00.675638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85" event={"ID":"01c0cedd-46c8-419d-b0fc-027b1d3d0b12","Type":"ContainerStarted","Data":"2313eb4408800236f3d3d2c92db3fbf98a778a29de70702e3cd662d83a6b8221"}
Sep 05 02:18:00.695113 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:00.695071    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jc85" podStartSLOduration=2.97859337 podStartE2EDuration="23.695058335s" totalImagesPullingTime="20.716464965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:00.694638293 +0000 UTC m=+24.666942170" watchObservedRunningTime="2026-09-05 02:18:00.695058335 +0000 UTC m=+24.667362212"
Sep 05 02:18:00.725898 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:00.725878    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:18:00.726403 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:00.726380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:18:01.545570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:01.545542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:01.545745 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:01.545648    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:18:01.677462 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:01.677433    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:18:01.677968 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:01.677948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tqks4"
Sep 05 02:18:02.545107 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.544933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:02.545205 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.545001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:02.545205 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:02.545182    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:18:02.545301 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:02.545276    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:18:02.682324 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.682256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" event={"ID":"a4d66a85-c9db-43a2-8465-62b8a630df97","Type":"ContainerStarted","Data":"8ca4fe8a861553598d3ac4932c6a52f71e091678516acccfd9966ddc6fee97e5"}
Sep 05 02:18:02.683106 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.682518    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:18:02.683106 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.682579    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:18:02.684191 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.684171    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5c424f-d9a6-4d3c-a4b1-c26857227b44" containerID="d93b972b2f898524787253ab97f6cbe11aa8f785518116b2e5de1e9054d45cb1" exitCode=0
Sep 05 02:18:02.684268 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.684246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerDied","Data":"d93b972b2f898524787253ab97f6cbe11aa8f785518116b2e5de1e9054d45cb1"}
Sep 05 02:18:02.697382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.697362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:18:02.713815 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:02.713779    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d" podStartSLOduration=7.181917526 podStartE2EDuration="25.713766757s" totalImagesPullingTime="18.531849231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:02.713737068 +0000 UTC m=+26.686040945" watchObservedRunningTime="2026-09-05 02:18:02.713766757 +0000 UTC m=+26.686070616"
Sep 05 02:18:03.544847 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:03.544820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:03.544981 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:03.544955    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:18:03.688973 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:03.688773    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5c424f-d9a6-4d3c-a4b1-c26857227b44" containerID="0ac8d35d0a24a78408b90beebd075594ac5bb04887e7bb6c1489de2cf9739263" exitCode=0
Sep 05 02:18:03.688973 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:03.688855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerDied","Data":"0ac8d35d0a24a78408b90beebd075594ac5bb04887e7bb6c1489de2cf9739263"}
Sep 05 02:18:03.689567 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:03.689527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:18:03.704433 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:03.704412    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:18:04.160119 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.160092    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c75c8"]
Sep 05 02:18:04.160277 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.160230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:04.160348 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:04.160323    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:18:04.162755 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.162728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-svblx"]
Sep 05 02:18:04.162866 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.162829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:04.162936 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:04.162918    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:18:04.165399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.165372    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk94l"]
Sep 05 02:18:04.165483 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.165465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:04.165581 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:04.165555    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:18:04.693346 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.693321    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5c424f-d9a6-4d3c-a4b1-c26857227b44" containerID="17ee386eaf44e55754e2da666348607e29a8254a2ffb17363de9c0c24de79dfa" exitCode=0
Sep 05 02:18:04.693632 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:04.693402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerDied","Data":"17ee386eaf44e55754e2da666348607e29a8254a2ffb17363de9c0c24de79dfa"}
Sep 05 02:18:05.545543 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:05.545513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:05.545543 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:05.545535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:05.545782 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:05.545620    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:18:05.545855 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:05.545781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:18:06.546014 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:06.545958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:06.546430 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:06.546094    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:18:07.544782 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:07.544751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:07.544937 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:07.544751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:07.544937 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:07.544888    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:18:07.545053 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:07.544954    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:18:08.545602 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:08.545570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:08.546017 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:08.545724    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75c8" podUID="4183dcc7-2263-45e0-97e8-ad25a6c7ad9a"
Sep 05 02:18:09.545313 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.545129    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:09.545446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.545183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:09.545446 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:09.545405    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk94l" podUID="4662252d-cd26-4d5c-ad62-0c815a71e462"
Sep 05 02:18:09.545532 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:09.545457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svblx" podUID="59e79702-89d7-49a0-bd00-8754b43eb10d"
Sep 05 02:18:09.708416 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.708392    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5c424f-d9a6-4d3c-a4b1-c26857227b44" containerID="f8d439e35a08d73185fa176234a5d8c47e59c387551d8114c981900db48e5ccf" exitCode=0
Sep 05 02:18:09.708716 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.708433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerDied","Data":"f8d439e35a08d73185fa176234a5d8c47e59c387551d8114c981900db48e5ccf"}
Sep 05 02:18:09.793826 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.793773    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-91.ec2.internal" event="NodeReady"
Sep 05 02:18:09.793915 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.793901    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 02:18:09.830398 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.830379    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57c954cd9c-kbj68"]
Sep 05 02:18:09.843005 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.842988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.845262 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.845236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 02:18:09.845370 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.845264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 02:18:09.845370 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.845263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jn8w8\""
Sep 05 02:18:09.845549 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.845536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 02:18:09.848266 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.848216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57c954cd9c-kbj68"]
Sep 05 02:18:09.853271 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.853254    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 02:18:09.854841 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.854822    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6b8vv"]
Sep 05 02:18:09.869164 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.869145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:09.869253 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.869152    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62k4n"]
Sep 05 02:18:09.873093 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.873074    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 02:18:09.873192 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.873164    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 02:18:09.873260 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.873241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l69pm\""
Sep 05 02:18:09.889541 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.889519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6b8vv"]
Sep 05 02:18:09.889647 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.889559    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62k4n"]
Sep 05 02:18:09.889647 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.889636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:09.893555 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.891831    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 02:18:09.893555 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.891845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2vlql\""
Sep 05 02:18:09.893555 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.892053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 02:18:09.893555 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.892058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 02:18:09.971206 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8nn4\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-kube-api-access-g8nn4\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.971276 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-config-volume\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:09.971276 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-tmp-dir\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:09.971351 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5f5z\" (UniqueName: \"kubernetes.io/projected/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-kube-api-access-h5f5z\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:09.971351 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-installation-pull-secrets\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.971351 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.971351 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e67d164f-d7b1-4a02-b7ae-c09ccc990765-ca-trust-extracted\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.971470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-bound-sa-token\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.971470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-certificates\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.971470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:09.971470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-image-registry-private-configuration\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:09.971470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:09.971453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-trusted-ca\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072011 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.071962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.072011 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.071987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-image-registry-private-configuration\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072116 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deeaa7ba-c6e9-461a-8163-c1d4d296f225-cert\") pod \"ingress-canary-62k4n\" (UID: \"deeaa7ba-c6e9-461a-8163-c1d4d296f225\") " pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:10.072116 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-trusted-ca\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072116 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qldz7\" (UniqueName: \"kubernetes.io/projected/deeaa7ba-c6e9-461a-8163-c1d4d296f225-kube-api-access-qldz7\") pod \"ingress-canary-62k4n\" (UID: \"deeaa7ba-c6e9-461a-8163-c1d4d296f225\") " pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:10.072216 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nn4\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-kube-api-access-g8nn4\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072216 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-config-volume\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.072216 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.072177    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:10.072355 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-tmp-dir\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.072355 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.072259    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls podName:7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.572239618 +0000 UTC m=+34.544543479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls") pod "dns-default-6b8vv" (UID: "7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:10.072456 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5f5z\" (UniqueName: \"kubernetes.io/projected/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-kube-api-access-h5f5z\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.072500 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-installation-pull-secrets\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072554 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-tmp-dir\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.072554 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072554 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e67d164f-d7b1-4a02-b7ae-c09ccc990765-ca-trust-extracted\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072736 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-bound-sa-token\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072736 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-certificates\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.072736 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-config-volume\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.072893 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.072821    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:10.072893 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.072836    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57c954cd9c-kbj68: secret "image-registry-tls" not found
Sep 05 02:18:10.072893 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.072887    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls podName:e67d164f-d7b1-4a02-b7ae-c09ccc990765 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:10.572872858 +0000 UTC m=+34.545176711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls") pod "image-registry-57c954cd9c-kbj68" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765") : secret "image-registry-tls" not found
Sep 05 02:18:10.073061 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.072953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e67d164f-d7b1-4a02-b7ae-c09ccc990765-ca-trust-extracted\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.073096 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.073079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-trusted-ca\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.073235 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.073213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-certificates\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.075988 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.075965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-installation-pull-secrets\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.076056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.076002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-image-registry-private-configuration\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.081045 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.081021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8nn4\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-kube-api-access-g8nn4\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.081141 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.081089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5f5z\" (UniqueName: \"kubernetes.io/projected/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-kube-api-access-h5f5z\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.082191 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.082167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-bound-sa-token\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.173399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.173378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:10.173463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.173405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:10.173463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.173430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:10.173463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.173462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deeaa7ba-c6e9-461a-8163-c1d4d296f225-cert\") pod \"ingress-canary-62k4n\" (UID: \"deeaa7ba-c6e9-461a-8163-c1d4d296f225\") " pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173507    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173529    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173551    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173567    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.173570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qldz7\" (UniqueName: \"kubernetes.io/projected/deeaa7ba-c6e9-461a-8163-c1d4d296f225-kube-api-access-qldz7\") pod \"ingress-canary-62k4n\" (UID: \"deeaa7ba-c6e9-461a-8163-c1d4d296f225\") " pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173580    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7z2j2 for pod openshift-network-diagnostics/network-check-target-svblx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173594    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret podName:4183dcc7-2263-45e0-97e8-ad25a6c7ad9a nodeName:}" failed. No retries permitted until 2026-09-05 02:18:42.173577669 +0000 UTC m=+66.145881530 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret") pod "global-pull-secret-syncer-c75c8" (UID: "4183dcc7-2263-45e0-97e8-ad25a6c7ad9a") : object "kube-system"/"original-pull-secret" not registered
Sep 05 02:18:10.173614 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173614    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:42.173604221 +0000 UTC m=+66.145908096 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 02:18:10.173949 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.173635    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2 podName:59e79702-89d7-49a0-bd00-8754b43eb10d nodeName:}" failed. No retries permitted until 2026-09-05 02:18:42.173623575 +0000 UTC m=+66.145927429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z2j2" (UniqueName: "kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2") pod "network-check-target-svblx" (UID: "59e79702-89d7-49a0-bd00-8754b43eb10d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 02:18:10.175370 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.175349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deeaa7ba-c6e9-461a-8163-c1d4d296f225-cert\") pod \"ingress-canary-62k4n\" (UID: \"deeaa7ba-c6e9-461a-8163-c1d4d296f225\") " pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:10.181928 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.181906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qldz7\" (UniqueName: \"kubernetes.io/projected/deeaa7ba-c6e9-461a-8163-c1d4d296f225-kube-api-access-qldz7\") pod \"ingress-canary-62k4n\" (UID: \"deeaa7ba-c6e9-461a-8163-c1d4d296f225\") " pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:10.239329 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.239302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62k4n"
Sep 05 02:18:10.544803 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.544773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:10.547289 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.547270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:10.550814 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.550790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62k4n"]
Sep 05 02:18:10.556113 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:18:10.556085    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeaa7ba_c6e9_461a_8163_c1d4d296f225.slice/crio-b0a99beec376cddd5b430c535c02aaec3eda88c553a04488065fc36af457f7dc WatchSource:0}: Error finding container b0a99beec376cddd5b430c535c02aaec3eda88c553a04488065fc36af457f7dc: Status 404 returned error can't find the container with id b0a99beec376cddd5b430c535c02aaec3eda88c553a04488065fc36af457f7dc
Sep 05 02:18:10.576968 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.576941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:10.577042 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.576973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:10.577089 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.577055    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:10.577089 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.577063    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:10.577089 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.577077    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57c954cd9c-kbj68: secret "image-registry-tls" not found
Sep 05 02:18:10.577192 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.577111    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls podName:7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:11.577094541 +0000 UTC m=+35.549398411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls") pod "dns-default-6b8vv" (UID: "7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:10.577192 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:10.577124    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls podName:e67d164f-d7b1-4a02-b7ae-c09ccc990765 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:11.577118981 +0000 UTC m=+35.549422835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls") pod "image-registry-57c954cd9c-kbj68" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765") : secret "image-registry-tls" not found
Sep 05 02:18:10.711485 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.711430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62k4n" event={"ID":"deeaa7ba-c6e9-461a-8163-c1d4d296f225","Type":"ContainerStarted","Data":"b0a99beec376cddd5b430c535c02aaec3eda88c553a04488065fc36af457f7dc"}
Sep 05 02:18:10.714408 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.714385    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb5c424f-d9a6-4d3c-a4b1-c26857227b44" containerID="70fad1d62ed8c40f590517546cee292d43751d0d3f408d13a7bb1c58e66b323a" exitCode=0
Sep 05 02:18:10.714491 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:10.714421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerDied","Data":"70fad1d62ed8c40f590517546cee292d43751d0d3f408d13a7bb1c58e66b323a"}
Sep 05 02:18:11.545500 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.545467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:11.545714 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.545467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:11.548693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.548649    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:11.548811 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.548717    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:11.548811 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.548650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4hwbs\""
Sep 05 02:18:11.548811 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.548652    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:11.548811 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.548650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66tlm\""
Sep 05 02:18:11.586008 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.585981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:11.586123 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.586024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:11.586186 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:11.586121    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:11.586186 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:11.586136    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57c954cd9c-kbj68: secret "image-registry-tls" not found
Sep 05 02:18:11.590055 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:11.586833    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:11.590055 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:11.586886    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls podName:e67d164f-d7b1-4a02-b7ae-c09ccc990765 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.586859618 +0000 UTC m=+37.559163490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls") pod "image-registry-57c954cd9c-kbj68" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765") : secret "image-registry-tls" not found
Sep 05 02:18:11.590055 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:11.586910    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls podName:7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:13.586899538 +0000 UTC m=+37.559203392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls") pod "dns-default-6b8vv" (UID: "7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:11.721416 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.721386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" event={"ID":"cb5c424f-d9a6-4d3c-a4b1-c26857227b44","Type":"ContainerStarted","Data":"13df8d024b1af55148903cd4c0c62b1024b7e05f029e03f5594c4b5b7029f8e8"}
Sep 05 02:18:11.743227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:11.743176    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7jm6h" podStartSLOduration=6.12434299 podStartE2EDuration="34.743160249s" totalImagesPullingTime="28.618817259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:11.742565089 +0000 UTC m=+35.714868965" watchObservedRunningTime="2026-09-05 02:18:11.743160249 +0000 UTC m=+35.715464125"
Sep 05 02:18:13.600127 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:13.600091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:13.600637 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:13.600147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:13.600637 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:13.600275    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:13.600637 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:13.600290    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57c954cd9c-kbj68: secret "image-registry-tls" not found
Sep 05 02:18:13.600637 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:13.600299    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:13.600637 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:13.600361    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls podName:7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:17.60034277 +0000 UTC m=+41.572646640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls") pod "dns-default-6b8vv" (UID: "7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:13.600637 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:13.600379    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls podName:e67d164f-d7b1-4a02-b7ae-c09ccc990765 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:17.600372877 +0000 UTC m=+41.572676731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls") pod "image-registry-57c954cd9c-kbj68" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765") : secret "image-registry-tls" not found
Sep 05 02:18:13.727615 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:13.727583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62k4n" event={"ID":"deeaa7ba-c6e9-461a-8163-c1d4d296f225","Type":"ContainerStarted","Data":"62e38566a996b440c2807e3023a15b80fdb527ad5e896a2c30ec5c5a0fe411d0"}
Sep 05 02:18:13.744189 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:13.743736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-62k4n" podStartSLOduration=2.23197281 podStartE2EDuration="4.743714679s" totalImagesPullingTime="2.511741869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:13.743203498 +0000 UTC m=+37.715507364" watchObservedRunningTime="2026-09-05 02:18:13.743714679 +0000 UTC m=+37.716018554"
Sep 05 02:18:17.628123 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:17.627922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:17.628606 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:17.628131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:17.628606 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:17.628068    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:17.628606 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:17.628206    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57c954cd9c-kbj68: secret "image-registry-tls" not found
Sep 05 02:18:17.628606 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:17.628222    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:17.628606 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:17.628283    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls podName:e67d164f-d7b1-4a02-b7ae-c09ccc990765 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:25.628259638 +0000 UTC m=+49.600563517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls") pod "image-registry-57c954cd9c-kbj68" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765") : secret "image-registry-tls" not found
Sep 05 02:18:17.628606 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:17.628303    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls podName:7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:25.628293885 +0000 UTC m=+49.600597743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls") pod "dns-default-6b8vv" (UID: "7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:25.685922 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:25.685878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:25.686441 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:25.685983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:25.686441 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:25.686053    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:25.686441 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:25.686072    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:25.686441 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:25.686083    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57c954cd9c-kbj68: secret "image-registry-tls" not found
Sep 05 02:18:25.686441 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:25.686133    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls podName:7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e nodeName:}" failed. No retries permitted until 2026-09-05 02:18:41.686108178 +0000 UTC m=+65.658412050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls") pod "dns-default-6b8vv" (UID: "7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:25.686441 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:25.686155    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls podName:e67d164f-d7b1-4a02-b7ae-c09ccc990765 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:41.686144867 +0000 UTC m=+65.658448730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls") pod "image-registry-57c954cd9c-kbj68" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765") : secret "image-registry-tls" not found
Sep 05 02:18:35.707114 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:35.707082    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnk6d"
Sep 05 02:18:41.697349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:41.697306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:18:41.697349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:41.697354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:18:41.697905 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:41.697486    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 02:18:41.697905 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:41.697498    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 05 02:18:41.697905 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:41.697527    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57c954cd9c-kbj68: secret "image-registry-tls" not found
Sep 05 02:18:41.697905 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:41.697553    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls podName:7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e nodeName:}" failed. No retries permitted until 2026-09-05 02:19:13.697533162 +0000 UTC m=+97.669837021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls") pod "dns-default-6b8vv" (UID: "7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e") : secret "dns-default-metrics-tls" not found
Sep 05 02:18:41.697905 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:41.697608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls podName:e67d164f-d7b1-4a02-b7ae-c09ccc990765 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:13.697587799 +0000 UTC m=+97.669891675 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls") pod "image-registry-57c954cd9c-kbj68" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765") : secret "image-registry-tls" not found
Sep 05 02:18:42.201871 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.201822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:18:42.201871 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.201873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:42.202158 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.201908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:42.204491 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.204463    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 02:18:42.204549 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.204467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 02:18:42.204549 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.204526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 02:18:42.212586 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:42.212556    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 02:18:42.212727 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:42.212640    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs podName:4662252d-cd26-4d5c-ad62-0c815a71e462 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:46.212618865 +0000 UTC m=+130.184922718 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs") pod "network-metrics-daemon-xk94l" (UID: "4662252d-cd26-4d5c-ad62-0c815a71e462") : secret "metrics-daemon-secret" not found
Sep 05 02:18:42.214782 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.214759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 02:18:42.216911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.216888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4183dcc7-2263-45e0-97e8-ad25a6c7ad9a-original-pull-secret\") pod \"global-pull-secret-syncer-c75c8\" (UID: \"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a\") " pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:42.225240 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.225216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2j2\" (UniqueName: \"kubernetes.io/projected/59e79702-89d7-49a0-bd00-8754b43eb10d-kube-api-access-7z2j2\") pod \"network-check-target-svblx\" (UID: \"59e79702-89d7-49a0-bd00-8754b43eb10d\") " pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:42.354841 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.354809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75c8"
Sep 05 02:18:42.459568 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.459507    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4hwbs\""
Sep 05 02:18:42.467685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.467652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:42.625900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.625865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c75c8"]
Sep 05 02:18:42.629177 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:18:42.629149    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4183dcc7_2263_45e0_97e8_ad25a6c7ad9a.slice/crio-d098e4a70eb8ac4297bf0f860a5271be7eefc8ff01c5a54e4ac5a29217559e75 WatchSource:0}: Error finding container d098e4a70eb8ac4297bf0f860a5271be7eefc8ff01c5a54e4ac5a29217559e75: Status 404 returned error can't find the container with id d098e4a70eb8ac4297bf0f860a5271be7eefc8ff01c5a54e4ac5a29217559e75
Sep 05 02:18:42.731449 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.731395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-svblx"]
Sep 05 02:18:42.734094 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:18:42.734069    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e79702_89d7_49a0_bd00_8754b43eb10d.slice/crio-76d5405ed125d2619d24736ec5b6e679dcba7b90d356d69ed279ae521b6e8225 WatchSource:0}: Error finding container 76d5405ed125d2619d24736ec5b6e679dcba7b90d356d69ed279ae521b6e8225: Status 404 returned error can't find the container with id 76d5405ed125d2619d24736ec5b6e679dcba7b90d356d69ed279ae521b6e8225
Sep 05 02:18:42.797089 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.797054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c75c8" event={"ID":"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a","Type":"ContainerStarted","Data":"d098e4a70eb8ac4297bf0f860a5271be7eefc8ff01c5a54e4ac5a29217559e75"}
Sep 05 02:18:42.798092 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:42.798056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-svblx" event={"ID":"59e79702-89d7-49a0-bd00-8754b43eb10d","Type":"ContainerStarted","Data":"76d5405ed125d2619d24736ec5b6e679dcba7b90d356d69ed279ae521b6e8225"}
Sep 05 02:18:44.843135 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.843105    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"]
Sep 05 02:18:44.845236 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.845213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:44.847580 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.847526    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 05 02:18:44.847986 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.847801    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 05 02:18:44.848207 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.848178    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 05 02:18:44.848207 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.848189    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d4b4s\""
Sep 05 02:18:44.857158 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.857139    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"]
Sep 05 02:18:44.920174 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.920146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:44.920288 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.920221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tmn\" (UniqueName: \"kubernetes.io/projected/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-kube-api-access-v6tmn\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:44.971319 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.971268    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-x7g2f"]
Sep 05 02:18:44.977741 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.977126    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"]
Sep 05 02:18:44.978688 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.978434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:44.981449 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.981361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-x7g2f"]
Sep 05 02:18:44.981547 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.981474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"]
Sep 05 02:18:44.981547 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.981502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:44.982822 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.982805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 02:18:44.983176 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.982810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 05 02:18:44.983783 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.983725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 05 02:18:44.983942 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.983879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4kb4t\""
Sep 05 02:18:44.984106 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.984069    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 02:18:44.984637 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.984617    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 02:18:44.984878 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.984859    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 05 02:18:44.984995 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.984979    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 02:18:44.985157 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.985137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6gcxj\""
Sep 05 02:18:44.985516 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.985155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 05 02:18:44.990702 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:44.990683    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 05 02:18:45.021522 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57df1726-1a5b-4ca3-828f-3c39ccbb533c-serving-cert\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.021624 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlg48\" (UniqueName: \"kubernetes.io/projected/57e56495-a3b3-4d73-b40d-001609204804-kube-api-access-mlg48\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.021624 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:45.021624 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/57df1726-1a5b-4ca3-828f-3c39ccbb533c-snapshots\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.021624 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57e56495-a3b3-4d73-b40d-001609204804-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.021854 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.021678    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:18:45.021854 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57df1726-1a5b-4ca3-828f-3c39ccbb533c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.021854 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.021733    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls podName:a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.521714578 +0000 UTC m=+69.494018447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cdsx4" (UID: "a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5") : secret "samples-operator-tls" not found
Sep 05 02:18:45.021854 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.021854 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tmn\" (UniqueName: \"kubernetes.io/projected/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-kube-api-access-v6tmn\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:45.021854 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57df1726-1a5b-4ca3-828f-3c39ccbb533c-tmp\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.022147 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.021901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57df1726-1a5b-4ca3-828f-3c39ccbb533c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.022147 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.022041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tggnv\" (UniqueName: \"kubernetes.io/projected/57df1726-1a5b-4ca3-828f-3c39ccbb533c-kube-api-access-tggnv\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.030850 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.030827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tmn\" (UniqueName: \"kubernetes.io/projected/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-kube-api-access-v6tmn\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:45.123002 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.122933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.123108 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.123092    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:45.123169 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.123143    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls podName:57e56495-a3b3-4d73-b40d-001609204804 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:45.623129376 +0000 UTC m=+69.595433247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tqkvz" (UID: "57e56495-a3b3-4d73-b40d-001609204804") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:45.123323 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57df1726-1a5b-4ca3-828f-3c39ccbb533c-tmp\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.123439 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57df1726-1a5b-4ca3-828f-3c39ccbb533c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.123439 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tggnv\" (UniqueName: \"kubernetes.io/projected/57df1726-1a5b-4ca3-828f-3c39ccbb533c-kube-api-access-tggnv\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.123544 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57df1726-1a5b-4ca3-828f-3c39ccbb533c-serving-cert\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.123544 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg48\" (UniqueName: \"kubernetes.io/projected/57e56495-a3b3-4d73-b40d-001609204804-kube-api-access-mlg48\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.123544 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/57df1726-1a5b-4ca3-828f-3c39ccbb533c-snapshots\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.123773 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57df1726-1a5b-4ca3-828f-3c39ccbb533c-tmp\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.124586 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.123991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57e56495-a3b3-4d73-b40d-001609204804-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.124586 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.124035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57df1726-1a5b-4ca3-828f-3c39ccbb533c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.124586 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.124097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/57df1726-1a5b-4ca3-828f-3c39ccbb533c-snapshots\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.124586 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.124339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57df1726-1a5b-4ca3-828f-3c39ccbb533c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.124586 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.124544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57df1726-1a5b-4ca3-828f-3c39ccbb533c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.124887 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.124656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/57e56495-a3b3-4d73-b40d-001609204804-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.126630 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.126531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57df1726-1a5b-4ca3-828f-3c39ccbb533c-serving-cert\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.133162 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.133112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlg48\" (UniqueName: \"kubernetes.io/projected/57e56495-a3b3-4d73-b40d-001609204804-kube-api-access-mlg48\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.133258 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.133206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tggnv\" (UniqueName: \"kubernetes.io/projected/57df1726-1a5b-4ca3-828f-3c39ccbb533c-kube-api-access-tggnv\") pod \"insights-operator-76bcd6ccf5-x7g2f\" (UID: \"57df1726-1a5b-4ca3-828f-3c39ccbb533c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.292749 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.292717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f"
Sep 05 02:18:45.528109 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.528077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:45.528269 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.528224    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:18:45.528327 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.528303    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls podName:a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:46.52828073 +0000 UTC m=+70.500584593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cdsx4" (UID: "a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5") : secret "samples-operator-tls" not found
Sep 05 02:18:45.629224 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.629187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:45.629369 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.629329    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:45.629429 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:45.629403    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls podName:57e56495-a3b3-4d73-b40d-001609204804 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:46.629382634 +0000 UTC m=+70.601686495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tqkvz" (UID: "57e56495-a3b3-4d73-b40d-001609204804") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:45.859510 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.859441    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8tdx7"]
Sep 05 02:18:45.861727 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.861700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7"
Sep 05 02:18:45.864087 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.864071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4k8ks\""
Sep 05 02:18:45.872562 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.872537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8tdx7"]
Sep 05 02:18:45.931800 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:45.931770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvg4\" (UniqueName: \"kubernetes.io/projected/79530a70-c1d2-4bf0-ad1c-a3c3af0cce86-kube-api-access-lsvg4\") pod \"network-check-source-685456655f-8tdx7\" (UID: \"79530a70-c1d2-4bf0-ad1c-a3c3af0cce86\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7"
Sep 05 02:18:46.032127 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:46.032093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvg4\" (UniqueName: \"kubernetes.io/projected/79530a70-c1d2-4bf0-ad1c-a3c3af0cce86-kube-api-access-lsvg4\") pod \"network-check-source-685456655f-8tdx7\" (UID: \"79530a70-c1d2-4bf0-ad1c-a3c3af0cce86\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7"
Sep 05 02:18:46.042478 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:46.042446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvg4\" (UniqueName: \"kubernetes.io/projected/79530a70-c1d2-4bf0-ad1c-a3c3af0cce86-kube-api-access-lsvg4\") pod \"network-check-source-685456655f-8tdx7\" (UID: \"79530a70-c1d2-4bf0-ad1c-a3c3af0cce86\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7"
Sep 05 02:18:46.171914 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:46.171847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7"
Sep 05 02:18:46.536013 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:46.535981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:46.536170 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:46.536145    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:18:46.536255 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:46.536220    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls podName:a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:48.536199776 +0000 UTC m=+72.508503651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cdsx4" (UID: "a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5") : secret "samples-operator-tls" not found
Sep 05 02:18:46.636975 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:46.636943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:46.637124 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:46.637107    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:46.637187 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:46.637170    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls podName:57e56495-a3b3-4d73-b40d-001609204804 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:48.637149504 +0000 UTC m=+72.609453390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tqkvz" (UID: "57e56495-a3b3-4d73-b40d-001609204804") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:47.554106 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.554083    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8tdx7"]
Sep 05 02:18:47.556254 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:18:47.556228    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79530a70_c1d2_4bf0_ad1c_a3c3af0cce86.slice/crio-4b1d738b642d81b208047243981f63ee005c133243f00d53a9d16db53b490700 WatchSource:0}: Error finding container 4b1d738b642d81b208047243981f63ee005c133243f00d53a9d16db53b490700: Status 404 returned error can't find the container with id 4b1d738b642d81b208047243981f63ee005c133243f00d53a9d16db53b490700
Sep 05 02:18:47.563194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.563174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-x7g2f"]
Sep 05 02:18:47.566766 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:18:47.566623    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57df1726_1a5b_4ca3_828f_3c39ccbb533c.slice/crio-a9be5ae92f434a5aca0ffffcb48feed74d3dd7b63f6d3b28453c110fc3c6a3d0 WatchSource:0}: Error finding container a9be5ae92f434a5aca0ffffcb48feed74d3dd7b63f6d3b28453c110fc3c6a3d0: Status 404 returned error can't find the container with id a9be5ae92f434a5aca0ffffcb48feed74d3dd7b63f6d3b28453c110fc3c6a3d0
Sep 05 02:18:47.814182 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.814114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c75c8" event={"ID":"4183dcc7-2263-45e0-97e8-ad25a6c7ad9a","Type":"ContainerStarted","Data":"17656eefa952731eb9448c655f6f322967f5fc88462fc791f9f835fdea8ee806"}
Sep 05 02:18:47.815675 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.815639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7" event={"ID":"79530a70-c1d2-4bf0-ad1c-a3c3af0cce86","Type":"ContainerStarted","Data":"5da1f8374d5ef4431975bb5d2c3ec19003c9f022d5098353b8d34c2b1f1b6b6c"}
Sep 05 02:18:47.815787 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.815689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7" event={"ID":"79530a70-c1d2-4bf0-ad1c-a3c3af0cce86","Type":"ContainerStarted","Data":"4b1d738b642d81b208047243981f63ee005c133243f00d53a9d16db53b490700"}
Sep 05 02:18:47.817107 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.817071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-svblx" event={"ID":"59e79702-89d7-49a0-bd00-8754b43eb10d","Type":"ContainerStarted","Data":"9796cd6ba8ef4364b307f5d9d8259f19eec0253603338a5013e418bb018671a4"}
Sep 05 02:18:47.817211 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.817194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:18:47.818248 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.818225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f" event={"ID":"57df1726-1a5b-4ca3-828f-3c39ccbb533c","Type":"ContainerStarted","Data":"a9be5ae92f434a5aca0ffffcb48feed74d3dd7b63f6d3b28453c110fc3c6a3d0"}
Sep 05 02:18:47.830094 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.830056    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c75c8" podStartSLOduration=66.165764513 podStartE2EDuration="1m10.830042441s" totalImagesPullingTime="4.664277928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:47.829199047 +0000 UTC m=+71.801502924" watchObservedRunningTime="2026-09-05 02:18:47.830042441 +0000 UTC m=+71.802346320"
Sep 05 02:18:47.845004 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.844970    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-8tdx7" podStartSLOduration=2.844958794 podStartE2EDuration="2.844958794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:47.8444769 +0000 UTC m=+71.816780775" watchObservedRunningTime="2026-09-05 02:18:47.844958794 +0000 UTC m=+71.817262708"
Sep 05 02:18:47.857916 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:47.857876    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-svblx" podStartSLOduration=66.306355295 podStartE2EDuration="1m10.857860115s" totalImagesPullingTime="4.55150482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:47.857507819 +0000 UTC m=+71.829811694" watchObservedRunningTime="2026-09-05 02:18:47.857860115 +0000 UTC m=+71.830163992"
Sep 05 02:18:48.551933 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:48.551909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:48.552089 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:48.552004    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:18:48.552089 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:48.552053    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls podName:a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.552038269 +0000 UTC m=+76.524342141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cdsx4" (UID: "a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5") : secret "samples-operator-tls" not found
Sep 05 02:18:48.652861 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:48.652833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:48.653256 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:48.652939    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:48.653256 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:48.652984    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls podName:57e56495-a3b3-4d73-b40d-001609204804 nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.652972463 +0000 UTC m=+76.625276317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tqkvz" (UID: "57e56495-a3b3-4d73-b40d-001609204804") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:49.824325 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:49.824294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f" event={"ID":"57df1726-1a5b-4ca3-828f-3c39ccbb533c","Type":"ContainerStarted","Data":"52cda29427527e946ccca07a0cd83bdd7fd6c9484a01fac871b35d0329c7bee6"}
Sep 05 02:18:49.841123 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:49.841075    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f" podStartSLOduration=3.659030126 podStartE2EDuration="5.841059984s" totalImagesPullingTime="2.182029858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:18:49.840645803 +0000 UTC m=+73.812949680" watchObservedRunningTime="2026-09-05 02:18:49.841059984 +0000 UTC m=+73.813363861"
Sep 05 02:18:51.473858 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.473817    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-glz8c"]
Sep 05 02:18:51.476988 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.476969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.479504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.479481    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 02:18:51.479617 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.479486    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 02:18:51.479617 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.479530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwbbc\""
Sep 05 02:18:51.494064 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.494042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glz8c"]
Sep 05 02:18:51.574551 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.574528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.574688 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.574573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cthb\" (UniqueName: \"kubernetes.io/projected/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-api-access-7cthb\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.574688 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.574591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.574818 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.574747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-crio-socket\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.574818 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.574790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-data-volume\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.675722 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.675697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-crio-socket\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.675797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.675726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-data-volume\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.675797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.675759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.675873 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.675815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-crio-socket\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.675873 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:51.675820    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:51.675930 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.675873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cthb\" (UniqueName: \"kubernetes.io/projected/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-api-access-7cthb\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.675930 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:51.675885    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.175871168 +0000 UTC m=+76.148175026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:51.675930 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.675918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.676048 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:51.676029    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:51.676082 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.676055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-data-volume\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:51.676112 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:51.676092    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:52.176076578 +0000 UTC m=+76.148380448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:51.690637 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:51.690615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cthb\" (UniqueName: \"kubernetes.io/projected/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-api-access-7cthb\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:52.179969 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:52.179938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:52.180077 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:52.179985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:52.180077 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.180059    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:52.180157 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.180091    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:52.180157 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.180122    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.180107952 +0000 UTC m=+77.152411812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:52.180157 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.180137    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:53.180131136 +0000 UTC m=+77.152434989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:52.583027 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:52.583003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:18:52.583380 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.583107    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 05 02:18:52.583380 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.583153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls podName:a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.583140742 +0000 UTC m=+84.555444596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cdsx4" (UID: "a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5") : secret "samples-operator-tls" not found
Sep 05 02:18:52.683411 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:52.683379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:18:52.683559 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.683534    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:52.683625 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:52.683611    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls podName:57e56495-a3b3-4d73-b40d-001609204804 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:00.683593622 +0000 UTC m=+84.655897478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tqkvz" (UID: "57e56495-a3b3-4d73-b40d-001609204804") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:18:53.186369 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:53.186335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:53.186501 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:53.186388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:53.186501 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:53.186464    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:53.186574 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:53.186506    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:53.186574 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:53.186533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:55.18651254 +0000 UTC m=+79.158816404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:53.186574 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:53.186551    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:55.186543092 +0000 UTC m=+79.158846945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:53.220736 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:53.220708    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6sbhk_a7eed417-1f97-47b2-b2e2-6ec960bbfda3/dns-node-resolver/0.log"
Sep 05 02:18:54.219363 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:54.219329    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2mzt5_d8f03596-1d37-470e-86e3-0b2ae5138bdf/node-ca/0.log"
Sep 05 02:18:55.200192 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:55.200155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:55.200394 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:55.200205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:55.200394 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:55.200294    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:55.200394 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:55.200323    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:55.200394 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:55.200371    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.200350094 +0000 UTC m=+83.172653967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : configmap "kube-rbac-proxy" not found
Sep 05 02:18:55.200394 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:55.200391    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:18:59.20038168 +0000 UTC m=+83.172685535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:55.219497 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:55.219474    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62k4n_deeaa7ba-c6e9-461a-8163-c1d4d296f225/serve-healthcheck-canary/0.log"
Sep 05 02:18:59.229928 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:59.229784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:59.229928 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:18:59.229835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:18:59.229928 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:59.229906    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 02:18:59.229928 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:59.229930    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.230518 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:59.229989    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:19:07.229969813 +0000 UTC m=+91.202273732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : secret "insights-runtime-extractor-tls" not found
Sep 05 02:18:59.230518 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:18:59.230019    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm podName:a1b9c92b-50cc-4a19-bc24-5643ce7aec0f nodeName:}" failed. No retries permitted until 2026-09-05 02:19:07.230009468 +0000 UTC m=+91.202313329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-glz8c" (UID: "a1b9c92b-50cc-4a19-bc24-5643ce7aec0f") : configmap "kube-rbac-proxy" not found
Sep 05 02:19:00.639789 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:00.639740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:19:00.642285 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:00.642260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cdsx4\" (UID: \"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:19:00.740984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:00.740955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:19:00.741122 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:00.741102    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:00.741189 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:00.741175    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls podName:57e56495-a3b3-4d73-b40d-001609204804 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:16.741157824 +0000 UTC m=+100.713461682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-tqkvz" (UID: "57e56495-a3b3-4d73-b40d-001609204804") : secret "cluster-monitoring-operator-tls" not found
Sep 05 02:19:00.757288 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:00.757264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"
Sep 05 02:19:01.033749 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:01.033720    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4"]
Sep 05 02:19:01.855820 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:01.855787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4" event={"ID":"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5","Type":"ContainerStarted","Data":"a131d3cf34b93cd7d0aa2c5e49a38a702d2ddcae2cef3dc590dacbc6659cee72"}
Sep 05 02:19:02.861518 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:02.861486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4" event={"ID":"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5","Type":"ContainerStarted","Data":"e61d957a415e677fd31fe748065597f2bfa1feba2040cc77905c12cd1cc0a3f8"}
Sep 05 02:19:03.867636 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:03.867594    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4" event={"ID":"a15c0a52-f2d5-4e02-b44f-8f5be73ed3d5","Type":"ContainerStarted","Data":"c07b84c1e5a8cd055235711f45b683f6aa665a14139262584ce9fcf556d74482"}
Sep 05 02:19:03.888651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:03.888602    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cdsx4" podStartSLOduration=18.191394747 podStartE2EDuration="19.88858685s" totalImagesPullingTime="1.697192103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:03.887357462 +0000 UTC m=+87.859661337" watchObservedRunningTime="2026-09-05 02:19:03.88858685 +0000 UTC m=+87.860890725"
Sep 05 02:19:07.295565 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.295507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:19:07.296020 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.295594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:19:07.296058 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.296037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:19:07.297810 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.297782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9c92b-50cc-4a19-bc24-5643ce7aec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-glz8c\" (UID: \"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f\") " pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:19:07.385528 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.385490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-glz8c"
Sep 05 02:19:07.656040 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.655897    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-glz8c"]
Sep 05 02:19:07.660881 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:07.660855    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b9c92b_50cc_4a19_bc24_5643ce7aec0f.slice/crio-b9dfd242f3de5dd732bca77eaa29faa3d07eb70456219b6b8ae00aa71beb20c7 WatchSource:0}: Error finding container b9dfd242f3de5dd732bca77eaa29faa3d07eb70456219b6b8ae00aa71beb20c7: Status 404 returned error can't find the container with id b9dfd242f3de5dd732bca77eaa29faa3d07eb70456219b6b8ae00aa71beb20c7
Sep 05 02:19:07.882371 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.882338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glz8c" event={"ID":"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f","Type":"ContainerStarted","Data":"5c94177a410af67a726debd1d300493175673079f65119ebdbf8155ca3c9cee8"}
Sep 05 02:19:07.882527 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:07.882394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glz8c" event={"ID":"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f","Type":"ContainerStarted","Data":"b9dfd242f3de5dd732bca77eaa29faa3d07eb70456219b6b8ae00aa71beb20c7"}
Sep 05 02:19:08.886959 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:08.886918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glz8c" event={"ID":"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f","Type":"ContainerStarted","Data":"5532311ea72ade8365ad4f0e1c174f8b4a8dcf12021a8b20cb9f8e976f8c1f8b"}
Sep 05 02:19:09.891381 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:09.891357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-glz8c" event={"ID":"a1b9c92b-50cc-4a19-bc24-5643ce7aec0f","Type":"ContainerStarted","Data":"33faee4ca09ea46f69d47eaa1310846a18b1819a656ea0abb1bf639af11b3107"}
Sep 05 02:19:09.915001 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:09.914959    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-glz8c" podStartSLOduration=16.869357294 podStartE2EDuration="18.914945908s" totalImagesPullingTime="2.045588614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:18:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:09.914782335 +0000 UTC m=+93.887086212" watchObservedRunningTime="2026-09-05 02:19:09.914945908 +0000 UTC m=+93.887249783"
Sep 05 02:19:13.741027 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.740983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:19:13.741485 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.741055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:19:13.743211 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.743188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e-metrics-tls\") pod \"dns-default-6b8vv\" (UID: \"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e\") " pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:19:13.743314 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.743296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"image-registry-57c954cd9c-kbj68\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") " pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:19:13.755790 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.755767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jn8w8\""
Sep 05 02:19:13.763704 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.763690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:19:13.841094 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.841062    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l69pm\""
Sep 05 02:19:13.845875 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:13.845850    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:19:14.064910 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.064860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57c954cd9c-kbj68"]
Sep 05 02:19:14.067479 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:14.067449    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67d164f_d7b1_4a02_b7ae_c09ccc990765.slice/crio-317bb4eb9cf3c18b1dc38ec9ba71fd88c7b53797ec3ec2e878fb1291cd714469 WatchSource:0}: Error finding container 317bb4eb9cf3c18b1dc38ec9ba71fd88c7b53797ec3ec2e878fb1291cd714469: Status 404 returned error can't find the container with id 317bb4eb9cf3c18b1dc38ec9ba71fd88c7b53797ec3ec2e878fb1291cd714469
Sep 05 02:19:14.111644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.111616    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"]
Sep 05 02:19:14.115517 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.115503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.117700 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.117658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 05 02:19:14.118233 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.118210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 02:19:14.118594 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.118577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 02:19:14.118676 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.118594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 02:19:14.120379 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.120360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mcgn7\""
Sep 05 02:19:14.128222 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.128203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6b8vv"]
Sep 05 02:19:14.154920 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:14.154890    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3623ea_47ad_4dd5_b6c6_9f892eb11c2e.slice/crio-4f65cff99b67ae9bc0f79c11c1980531523794a385556ae4c24cb017ed65918d WatchSource:0}: Error finding container 4f65cff99b67ae9bc0f79c11c1980531523794a385556ae4c24cb017ed65918d: Status 404 returned error can't find the container with id 4f65cff99b67ae9bc0f79c11c1980531523794a385556ae4c24cb017ed65918d
Sep 05 02:19:14.158727 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.158686    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"]
Sep 05 02:19:14.222157 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.222124    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-8wk42"]
Sep 05 02:19:14.225408 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.225393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-8wk42"
Sep 05 02:19:14.229591 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.229569    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 05 02:19:14.230382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.230363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 05 02:19:14.230484 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.230363    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bs4k5\""
Sep 05 02:19:14.237790 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.237769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-8wk42"]
Sep 05 02:19:14.244990 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.244969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccbc6477-6030-440a-8f95-0b4d33cd47bd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b69767bf8-v7v4w\" (UID: \"ccbc6477-6030-440a-8f95-0b4d33cd47bd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.245069 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.245006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42qp8\" (UniqueName: \"kubernetes.io/projected/ccbc6477-6030-440a-8f95-0b4d33cd47bd-kube-api-access-42qp8\") pod \"managed-serviceaccount-addon-agent-6b69767bf8-v7v4w\" (UID: \"ccbc6477-6030-440a-8f95-0b4d33cd47bd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.345795 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.345712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x569\" (UniqueName: \"kubernetes.io/projected/4bf3cac2-d91a-49b2-9922-8484815eaf7d-kube-api-access-9x569\") pod \"downloads-85d98c998f-8wk42\" (UID: \"4bf3cac2-d91a-49b2-9922-8484815eaf7d\") " pod="openshift-console/downloads-85d98c998f-8wk42"
Sep 05 02:19:14.345795 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.345783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccbc6477-6030-440a-8f95-0b4d33cd47bd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b69767bf8-v7v4w\" (UID: \"ccbc6477-6030-440a-8f95-0b4d33cd47bd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.346015 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.345845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42qp8\" (UniqueName: \"kubernetes.io/projected/ccbc6477-6030-440a-8f95-0b4d33cd47bd-kube-api-access-42qp8\") pod \"managed-serviceaccount-addon-agent-6b69767bf8-v7v4w\" (UID: \"ccbc6477-6030-440a-8f95-0b4d33cd47bd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.348246 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.348227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ccbc6477-6030-440a-8f95-0b4d33cd47bd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b69767bf8-v7v4w\" (UID: \"ccbc6477-6030-440a-8f95-0b4d33cd47bd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.361615 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.361585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42qp8\" (UniqueName: \"kubernetes.io/projected/ccbc6477-6030-440a-8f95-0b4d33cd47bd-kube-api-access-42qp8\") pod \"managed-serviceaccount-addon-agent-6b69767bf8-v7v4w\" (UID: \"ccbc6477-6030-440a-8f95-0b4d33cd47bd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.435836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.435806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"
Sep 05 02:19:14.447061 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.447035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x569\" (UniqueName: \"kubernetes.io/projected/4bf3cac2-d91a-49b2-9922-8484815eaf7d-kube-api-access-9x569\") pod \"downloads-85d98c998f-8wk42\" (UID: \"4bf3cac2-d91a-49b2-9922-8484815eaf7d\") " pod="openshift-console/downloads-85d98c998f-8wk42"
Sep 05 02:19:14.461314 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.461288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x569\" (UniqueName: \"kubernetes.io/projected/4bf3cac2-d91a-49b2-9922-8484815eaf7d-kube-api-access-9x569\") pod \"downloads-85d98c998f-8wk42\" (UID: \"4bf3cac2-d91a-49b2-9922-8484815eaf7d\") " pod="openshift-console/downloads-85d98c998f-8wk42"
Sep 05 02:19:14.534214 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.534179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-8wk42"
Sep 05 02:19:14.707834 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.707763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w"]
Sep 05 02:19:14.711254 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:14.711222    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccbc6477_6030_440a_8f95_0b4d33cd47bd.slice/crio-b712c324b746ccf4623808e4ca2503f78771e40a85834b807d1f310319a05ad5 WatchSource:0}: Error finding container b712c324b746ccf4623808e4ca2503f78771e40a85834b807d1f310319a05ad5: Status 404 returned error can't find the container with id b712c324b746ccf4623808e4ca2503f78771e40a85834b807d1f310319a05ad5
Sep 05 02:19:14.806951 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.806901    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-8wk42"]
Sep 05 02:19:14.811147 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:14.811118    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf3cac2_d91a_49b2_9922_8484815eaf7d.slice/crio-a9f8b83ffd709a406cfa31300def859fc53ff4eb97cc1e02800186eb56cbc50b WatchSource:0}: Error finding container a9f8b83ffd709a406cfa31300def859fc53ff4eb97cc1e02800186eb56cbc50b: Status 404 returned error can't find the container with id a9f8b83ffd709a406cfa31300def859fc53ff4eb97cc1e02800186eb56cbc50b
Sep 05 02:19:14.908353 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.908314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-8wk42" event={"ID":"4bf3cac2-d91a-49b2-9922-8484815eaf7d","Type":"ContainerStarted","Data":"a9f8b83ffd709a406cfa31300def859fc53ff4eb97cc1e02800186eb56cbc50b"}
Sep 05 02:19:14.909540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.909509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w" event={"ID":"ccbc6477-6030-440a-8f95-0b4d33cd47bd","Type":"ContainerStarted","Data":"b712c324b746ccf4623808e4ca2503f78771e40a85834b807d1f310319a05ad5"}
Sep 05 02:19:14.910813 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.910790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6b8vv" event={"ID":"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e","Type":"ContainerStarted","Data":"4f65cff99b67ae9bc0f79c11c1980531523794a385556ae4c24cb017ed65918d"}
Sep 05 02:19:14.912316 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.912290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68" event={"ID":"e67d164f-d7b1-4a02-b7ae-c09ccc990765","Type":"ContainerStarted","Data":"4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c"}
Sep 05 02:19:14.912421 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.912332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68" event={"ID":"e67d164f-d7b1-4a02-b7ae-c09ccc990765","Type":"ContainerStarted","Data":"317bb4eb9cf3c18b1dc38ec9ba71fd88c7b53797ec3ec2e878fb1291cd714469"}
Sep 05 02:19:14.912740 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.912720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:19:14.937391 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:14.937345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68" podStartSLOduration=76.937329451 podStartE2EDuration="1m16.937329451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:14.936345206 +0000 UTC m=+98.908649081" watchObservedRunningTime="2026-09-05 02:19:14.937329451 +0000 UTC m=+98.909633326"
Sep 05 02:19:16.766525 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:16.766482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:19:16.769199 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:16.769164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e56495-a3b3-4d73-b40d-001609204804-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-tqkvz\" (UID: \"57e56495-a3b3-4d73-b40d-001609204804\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:19:16.799126 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:16.799097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"
Sep 05 02:19:16.923289 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:16.923254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6b8vv" event={"ID":"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e","Type":"ContainerStarted","Data":"5b9ff9d4539d8f593ebd187c145c11d7be64baf34c58e7a9ee8c85955f1a883f"}
Sep 05 02:19:16.923464 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:16.923342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6b8vv" event={"ID":"7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e","Type":"ContainerStarted","Data":"b215799edee660226cf97bf0baccd6326316322c849865a943dcca71815d8b2d"}
Sep 05 02:19:16.923526 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:16.923464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:19:16.946443 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:16.946378    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6b8vv" podStartSLOduration=66.223819376 podStartE2EDuration="1m7.946363509s" totalImagesPullingTime="1.722544133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:16.944239953 +0000 UTC m=+100.916543831" watchObservedRunningTime="2026-09-05 02:19:16.946363509 +0000 UTC m=+100.918667385"
Sep 05 02:19:17.074704 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:17.074611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz"]
Sep 05 02:19:17.078358 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:17.078318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e56495_a3b3_4d73_b40d_001609204804.slice/crio-3bd3140885b925d396d733e15fa97002a8c1e4813eb8abcd8d016f495b920fc5 WatchSource:0}: Error finding container 3bd3140885b925d396d733e15fa97002a8c1e4813eb8abcd8d016f495b920fc5: Status 404 returned error can't find the container with id 3bd3140885b925d396d733e15fa97002a8c1e4813eb8abcd8d016f495b920fc5
Sep 05 02:19:17.931194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:17.931114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz" event={"ID":"57e56495-a3b3-4d73-b40d-001609204804","Type":"ContainerStarted","Data":"3bd3140885b925d396d733e15fa97002a8c1e4813eb8abcd8d016f495b920fc5"}
Sep 05 02:19:18.823427 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:18.823392    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-svblx"
Sep 05 02:19:19.385480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.385440    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dccf85fb9-2x9mr"]
Sep 05 02:19:19.413207 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.413175    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dccf85fb9-2x9mr"]
Sep 05 02:19:19.413372 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.413285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.415803 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.415741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 05 02:19:19.416375 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.416054    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 05 02:19:19.416375 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.416051    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 05 02:19:19.416375 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.416191    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 05 02:19:19.416641 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.416433    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 05 02:19:19.416641 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.416609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r5f89\""
Sep 05 02:19:19.589142 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.589107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnwl\" (UniqueName: \"kubernetes.io/projected/dfe2e105-5f18-4566-958d-006a467010a0-kube-api-access-mvnwl\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.589318 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.589301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-serving-cert\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.589364 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.589332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-console-config\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.589435 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.589408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-oauth-config\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.589487 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.589470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-service-ca\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.589559 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.589540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-oauth-serving-cert\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.690724 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.690630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-serving-cert\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.690883 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.690725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-console-config\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.690883 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.690766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-oauth-config\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.690883 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.690793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-service-ca\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.690883 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.690836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-oauth-serving-cert\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.690883 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.690864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnwl\" (UniqueName: \"kubernetes.io/projected/dfe2e105-5f18-4566-958d-006a467010a0-kube-api-access-mvnwl\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.691431 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.691401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-console-config\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.691753 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.691728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-service-ca\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.691952 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.691905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-oauth-serving-cert\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.694136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.694112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-oauth-config\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.694981 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.694940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-serving-cert\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.699200 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.699157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnwl\" (UniqueName: \"kubernetes.io/projected/dfe2e105-5f18-4566-958d-006a467010a0-kube-api-access-mvnwl\") pod \"console-7dccf85fb9-2x9mr\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") " pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:19.725921 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:19.725885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:20.901091 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:20.900677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dccf85fb9-2x9mr"]
Sep 05 02:19:20.906198 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:20.906160    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe2e105_5f18_4566_958d_006a467010a0.slice/crio-620a6e0d791357617f7055aa83572fac0e3af1c308452d73623ce0850caf8c89 WatchSource:0}: Error finding container 620a6e0d791357617f7055aa83572fac0e3af1c308452d73623ce0850caf8c89: Status 404 returned error can't find the container with id 620a6e0d791357617f7055aa83572fac0e3af1c308452d73623ce0850caf8c89
Sep 05 02:19:20.944434 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:20.944183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w" event={"ID":"ccbc6477-6030-440a-8f95-0b4d33cd47bd","Type":"ContainerStarted","Data":"c1d82afeadff8a9400f9919d8560e9acc23f9aac5fed5d26db4a9b0e14cd5d83"}
Sep 05 02:19:20.946001 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:20.945980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz" event={"ID":"57e56495-a3b3-4d73-b40d-001609204804","Type":"ContainerStarted","Data":"a0a45a911d230e8ed405085155e653a399dd1dd3cf3bac7f6ba2e6c275a45053"}
Sep 05 02:19:20.947309 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:20.947284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccf85fb9-2x9mr" event={"ID":"dfe2e105-5f18-4566-958d-006a467010a0","Type":"ContainerStarted","Data":"620a6e0d791357617f7055aa83572fac0e3af1c308452d73623ce0850caf8c89"}
Sep 05 02:19:20.966241 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:20.966200    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b69767bf8-v7v4w" podStartSLOduration=1.050334468 podStartE2EDuration="6.966184824s" totalImagesPullingTime="5.915850356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:20.964975693 +0000 UTC m=+104.937279570" watchObservedRunningTime="2026-09-05 02:19:20.966184824 +0000 UTC m=+104.938488723"
Sep 05 02:19:20.986003 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:20.985966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-tqkvz" podStartSLOduration=33.433241195 podStartE2EDuration="36.985952126s" totalImagesPullingTime="3.552710931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:20.985539099 +0000 UTC m=+104.957842973" watchObservedRunningTime="2026-09-05 02:19:20.985952126 +0000 UTC m=+104.958256001"
Sep 05 02:19:21.248056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.247974    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"]
Sep 05 02:19:21.271549 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.271517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"]
Sep 05 02:19:21.271729 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.271624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:21.274339 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.274314    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 05 02:19:21.274474 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.274394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-phgj7\""
Sep 05 02:19:21.305713 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.305655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wk789\" (UID: \"1bb1005d-8930-4403-bee7-7bcb5475deab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:21.406454 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.406420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wk789\" (UID: \"1bb1005d-8930-4403-bee7-7bcb5475deab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:21.406623 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:21.406588    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 05 02:19:21.406704 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:21.406654    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates podName:1bb1005d-8930-4403-bee7-7bcb5475deab nodeName:}" failed. No retries permitted until 2026-09-05 02:19:21.906639539 +0000 UTC m=+105.878943410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-wk789" (UID: "1bb1005d-8930-4403-bee7-7bcb5475deab") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 05 02:19:21.909447 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:21.909407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wk789\" (UID: \"1bb1005d-8930-4403-bee7-7bcb5475deab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:21.909990 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:21.909532    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 05 02:19:21.909990 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:21.909603    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates podName:1bb1005d-8930-4403-bee7-7bcb5475deab nodeName:}" failed. No retries permitted until 2026-09-05 02:19:22.909582572 +0000 UTC m=+106.881886446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-wk789" (UID: "1bb1005d-8930-4403-bee7-7bcb5475deab") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 05 02:19:22.917619 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:22.917497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wk789\" (UID: \"1bb1005d-8930-4403-bee7-7bcb5475deab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:22.920719 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:22.920645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1bb1005d-8930-4403-bee7-7bcb5475deab-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-wk789\" (UID: \"1bb1005d-8930-4403-bee7-7bcb5475deab\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:23.083461 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:23.083410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:24.333040 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:24.332946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"]
Sep 05 02:19:24.336501 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:24.336432    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb1005d_8930_4403_bee7_7bcb5475deab.slice/crio-66a5809d25d93edf583fb6e4650b61c0dd0cce9105df2ad575359e30b577325f WatchSource:0}: Error finding container 66a5809d25d93edf583fb6e4650b61c0dd0cce9105df2ad575359e30b577325f: Status 404 returned error can't find the container with id 66a5809d25d93edf583fb6e4650b61c0dd0cce9105df2ad575359e30b577325f
Sep 05 02:19:24.964444 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:24.964053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccf85fb9-2x9mr" event={"ID":"dfe2e105-5f18-4566-958d-006a467010a0","Type":"ContainerStarted","Data":"0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f"}
Sep 05 02:19:24.965443 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:24.965407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789" event={"ID":"1bb1005d-8930-4403-bee7-7bcb5475deab","Type":"ContainerStarted","Data":"66a5809d25d93edf583fb6e4650b61c0dd0cce9105df2ad575359e30b577325f"}
Sep 05 02:19:24.983318 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:24.983266    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dccf85fb9-2x9mr" podStartSLOduration=2.79954312 podStartE2EDuration="5.983249124s" totalImagesPullingTime="3.183706004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:24.981567187 +0000 UTC m=+108.953871062" watchObservedRunningTime="2026-09-05 02:19:24.983249124 +0000 UTC m=+108.955552991"
Sep 05 02:19:26.933801 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:26.933768    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6b8vv"
Sep 05 02:19:28.903863 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.903828    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f67c845cd-tdvwv"]
Sep 05 02:19:28.907271 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.907250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:28.914629 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.914600    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 05 02:19:28.916204 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.916180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f67c845cd-tdvwv"]
Sep 05 02:19:28.968786 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.968753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-oauth-serving-cert\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:28.968960 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.968827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-config\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:28.968960 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.968855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-trusted-ca-bundle\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:28.968960 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.968885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-serving-cert\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:28.969136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.968997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-oauth-config\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:28.969136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.969023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-service-ca\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:28.969136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:28.969101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcdt\" (UniqueName: \"kubernetes.io/projected/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-kube-api-access-qfcdt\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.069963 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.069924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-config\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.070141 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.069989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-trusted-ca-bundle\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.070141 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.070026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-serving-cert\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.070141 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.070053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-oauth-config\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.070334 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.070068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-service-ca\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.070543 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.070522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcdt\" (UniqueName: \"kubernetes.io/projected/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-kube-api-access-qfcdt\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.070628 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.070591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-oauth-serving-cert\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.072369 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.071399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-oauth-serving-cert\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.072369 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.071993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-service-ca\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.073139 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.072889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-config\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.076342 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.073702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-trusted-ca-bundle\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.076342 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.075059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-oauth-config\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.077561 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.077541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-serving-cert\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.081868 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.081841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcdt\" (UniqueName: \"kubernetes.io/projected/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-kube-api-access-qfcdt\") pod \"console-6f67c845cd-tdvwv\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") " pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.219448 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.219358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:29.726646 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.726599    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:29.726863 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.726722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:29.732250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.732221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:29.990295 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:29.990216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:19:31.819653 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:31.819623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f67c845cd-tdvwv"]
Sep 05 02:19:31.823652 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:31.822356    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca66ac21_f7f0_45ce_a1a0_296e35f2020a.slice/crio-80ab1624913170ac1b803d0a75356d38e8d9bc0bfb1737bb147d55d5e48054b8 WatchSource:0}: Error finding container 80ab1624913170ac1b803d0a75356d38e8d9bc0bfb1737bb147d55d5e48054b8: Status 404 returned error can't find the container with id 80ab1624913170ac1b803d0a75356d38e8d9bc0bfb1737bb147d55d5e48054b8
Sep 05 02:19:31.997634 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:31.997517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f67c845cd-tdvwv" event={"ID":"ca66ac21-f7f0-45ce-a1a0-296e35f2020a","Type":"ContainerStarted","Data":"f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18"}
Sep 05 02:19:31.997634 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:31.997587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f67c845cd-tdvwv" event={"ID":"ca66ac21-f7f0-45ce-a1a0-296e35f2020a","Type":"ContainerStarted","Data":"80ab1624913170ac1b803d0a75356d38e8d9bc0bfb1737bb147d55d5e48054b8"}
Sep 05 02:19:32.001421 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:32.001351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-8wk42" event={"ID":"4bf3cac2-d91a-49b2-9922-8484815eaf7d","Type":"ContainerStarted","Data":"c82facd6836f95056eb99bf6c1bf98f1e1bd1aae11a0edf9d15379caf743fd15"}
Sep 05 02:19:32.001859 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:32.001812    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-8wk42"
Sep 05 02:19:32.017153 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:32.017051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-8wk42"
Sep 05 02:19:32.032844 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:32.032601    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f67c845cd-tdvwv" podStartSLOduration=4.032580653 podStartE2EDuration="4.032580653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:32.03133809 +0000 UTC m=+116.003641966" watchObservedRunningTime="2026-09-05 02:19:32.032580653 +0000 UTC m=+116.004884531"
Sep 05 02:19:32.070927 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:32.069899    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-8wk42" podStartSLOduration=1.244033591 podStartE2EDuration="18.069882455s" totalImagesPullingTime="16.825848864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:32.069752374 +0000 UTC m=+116.042056251" watchObservedRunningTime="2026-09-05 02:19:32.069882455 +0000 UTC m=+116.042186350"
Sep 05 02:19:33.007344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:33.007301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789" event={"ID":"1bb1005d-8930-4403-bee7-7bcb5475deab","Type":"ContainerStarted","Data":"ebf48ca41e763fd9c1990d8799c5b937767cd4c4c7b425ab073943f685fee395"}
Sep 05 02:19:33.007841 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:33.007686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:33.013264 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:33.013235    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789"
Sep 05 02:19:33.028836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:33.028780    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-wk789" podStartSLOduration=3.96309069 podStartE2EDuration="12.028764242s" totalImagesPullingTime="8.065673552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:33.027683894 +0000 UTC m=+116.999987764" watchObservedRunningTime="2026-09-05 02:19:33.028764242 +0000 UTC m=+117.001068118"
Sep 05 02:19:35.922142 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:35.922111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:19:36.666413 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:36.666379    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57c954cd9c-kbj68"]
Sep 05 02:19:39.220413 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.220366    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:39.220985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.220451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:39.226142 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.226108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:39.758732 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.758694    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"]
Sep 05 02:19:39.810010 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.809977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"]
Sep 05 02:19:39.810185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.810037    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x57pm"]
Sep 05 02:19:39.810255 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.810208    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.815624 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.815598    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:19:39.815789 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.815621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 05 02:19:39.815789 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.815677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 02:19:39.823757 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.823732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7j8lf\""
Sep 05 02:19:39.836868 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.836834    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-242ct"]
Sep 05 02:19:39.836999 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.836979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.845428 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.845404    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 02:19:39.845796 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.845775    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 02:19:39.845898 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.845808    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4v597\""
Sep 05 02:19:39.846261 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.846244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 02:19:39.855059 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.855034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-242ct"]
Sep 05 02:19:39.855193 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.855173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:39.860257 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.860233    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 05 02:19:39.860362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.860262    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 05 02:19:39.860362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.860339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-k5575\""
Sep 05 02:19:39.860527 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.860502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 05 02:19:39.862962 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.862937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4jv7\" (UniqueName: \"kubernetes.io/projected/d9a4adad-8278-4253-a58e-a20c3f4464f6-kube-api-access-z4jv7\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.863077 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.863021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.863132 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.863096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9a4adad-8278-4253-a58e-a20c3f4464f6-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.863181 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.863150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.964448 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:39.964448 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4jv7\" (UniqueName: \"kubernetes.io/projected/d9a4adad-8278-4253-a58e-a20c3f4464f6-kube-api-access-z4jv7\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.964721 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-root\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.964721 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d817548-f439-4d19-b3c5-c74c8a1c92f8-metrics-client-ca\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.964721 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/30d67f0e-58fc-46f2-8e09-d68e9a20d747-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:39.964721 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:39.964721 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-tls\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7dm\" (UniqueName: \"kubernetes.io/projected/5d817548-f439-4d19-b3c5-c74c8a1c92f8-kube-api-access-2j7dm\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlsbh\" (UniqueName: \"kubernetes.io/projected/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-api-access-qlsbh\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9a4adad-8278-4253-a58e-a20c3f4464f6-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-wtmp\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:39.964979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-sys\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.965361 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.964984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.965361 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.965009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d67f0e-58fc-46f2-8e09-d68e9a20d747-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:39.965361 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.965035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-textfile\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.965361 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.965072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:39.965361 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:39.965091    2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 05 02:19:39.965361 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:39.965192    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-tls podName:d9a4adad-8278-4253-a58e-a20c3f4464f6 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:40.465158796 +0000 UTC m=+124.437462663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-w6f6b" (UID: "d9a4adad-8278-4253-a58e-a20c3f4464f6") : secret "openshift-state-metrics-tls" not found
Sep 05 02:19:39.966150 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.966109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9a4adad-8278-4253-a58e-a20c3f4464f6-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.971705 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.971657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:39.974335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:39.974314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4jv7\" (UniqueName: \"kubernetes.io/projected/d9a4adad-8278-4253-a58e-a20c3f4464f6-kube-api-access-z4jv7\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:40.040841 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.040769    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:19:40.066143 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlsbh\" (UniqueName: \"kubernetes.io/projected/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-api-access-qlsbh\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.066310 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066376 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-wtmp\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066437 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.066437 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-sys\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066538 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d67f0e-58fc-46f2-8e09-d68e9a20d747-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.066538 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-textfile\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066636 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-wtmp\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066636 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066636 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-sys\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.066828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-root\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d817548-f439-4d19-b3c5-c74c8a1c92f8-metrics-client-ca\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.066828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/30d67f0e-58fc-46f2-8e09-d68e9a20d747-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.066828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.067063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-tls\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.067063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7dm\" (UniqueName: \"kubernetes.io/projected/5d817548-f439-4d19-b3c5-c74c8a1c92f8-kube-api-access-2j7dm\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.067063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.066889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-textfile\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.067063 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:40.066999    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 05 02:19:40.067259 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:40.067068    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-tls podName:5d817548-f439-4d19-b3c5-c74c8a1c92f8 nodeName:}" failed. No retries permitted until 2026-09-05 02:19:40.567048311 +0000 UTC m=+124.539352165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-tls") pod "node-exporter-x57pm" (UID: "5d817548-f439-4d19-b3c5-c74c8a1c92f8") : secret "node-exporter-tls" not found
Sep 05 02:19:40.067259 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.067064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d817548-f439-4d19-b3c5-c74c8a1c92f8-root\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.067822 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.067795    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d817548-f439-4d19-b3c5-c74c8a1c92f8-metrics-client-ca\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.068065 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.067860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.068638 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.068591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.068638 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.068603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d67f0e-58fc-46f2-8e09-d68e9a20d747-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.069625 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.069600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/30d67f0e-58fc-46f2-8e09-d68e9a20d747-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.070399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.070333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.070892 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.070873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.071472 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.071453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.091117 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.091064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7dm\" (UniqueName: \"kubernetes.io/projected/5d817548-f439-4d19-b3c5-c74c8a1c92f8-kube-api-access-2j7dm\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.094942 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.094910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlsbh\" (UniqueName: \"kubernetes.io/projected/30d67f0e-58fc-46f2-8e09-d68e9a20d747-kube-api-access-qlsbh\") pod \"kube-state-metrics-74cb57df7-242ct\" (UID: \"30d67f0e-58fc-46f2-8e09-d68e9a20d747\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.116777 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.116741    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dccf85fb9-2x9mr"]
Sep 05 02:19:40.165040 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.165006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct"
Sep 05 02:19:40.470278 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.470193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:40.473417 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.473388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9a4adad-8278-4253-a58e-a20c3f4464f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-w6f6b\" (UID: \"d9a4adad-8278-4253-a58e-a20c3f4464f6\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:40.496551 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.496521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-242ct"]
Sep 05 02:19:40.497004 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:40.496975    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d67f0e_58fc_46f2_8e09_d68e9a20d747.slice/crio-02059d8179ebda35268407ab8b36246ff3bbc63fa23a2cff5d2da665200b651a WatchSource:0}: Error finding container 02059d8179ebda35268407ab8b36246ff3bbc63fa23a2cff5d2da665200b651a: Status 404 returned error can't find the container with id 02059d8179ebda35268407ab8b36246ff3bbc63fa23a2cff5d2da665200b651a
Sep 05 02:19:40.571183 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.571148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-tls\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.573685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.573643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d817548-f439-4d19-b3c5-c74c8a1c92f8-node-exporter-tls\") pod \"node-exporter-x57pm\" (UID: \"5d817548-f439-4d19-b3c5-c74c8a1c92f8\") " pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.721957 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.721872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"
Sep 05 02:19:40.748380 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:40.748344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x57pm"
Sep 05 02:19:40.789318 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:40.789277    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d817548_f439_4d19_b3c5_c74c8a1c92f8.slice/crio-6b59ac234f18183126367302671555958a588b35b0938fb7755e0614d535957f WatchSource:0}: Error finding container 6b59ac234f18183126367302671555958a588b35b0938fb7755e0614d535957f: Status 404 returned error can't find the container with id 6b59ac234f18183126367302671555958a588b35b0938fb7755e0614d535957f
Sep 05 02:19:41.039124 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:41.038986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b"]
Sep 05 02:19:41.041938 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:41.041910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct" event={"ID":"30d67f0e-58fc-46f2-8e09-d68e9a20d747","Type":"ContainerStarted","Data":"02059d8179ebda35268407ab8b36246ff3bbc63fa23a2cff5d2da665200b651a"}
Sep 05 02:19:41.042281 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:41.042251    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a4adad_8278_4253_a58e_a20c3f4464f6.slice/crio-def199ff8446aea53f0dc0de2967e488376282a83e8dac54c6b3b84b7f13f41e WatchSource:0}: Error finding container def199ff8446aea53f0dc0de2967e488376282a83e8dac54c6b3b84b7f13f41e: Status 404 returned error can't find the container with id def199ff8446aea53f0dc0de2967e488376282a83e8dac54c6b3b84b7f13f41e
Sep 05 02:19:41.044032 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:41.044000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x57pm" event={"ID":"5d817548-f439-4d19-b3c5-c74c8a1c92f8","Type":"ContainerStarted","Data":"6b59ac234f18183126367302671555958a588b35b0938fb7755e0614d535957f"}
Sep 05 02:19:42.048972 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:42.048941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b" event={"ID":"d9a4adad-8278-4253-a58e-a20c3f4464f6","Type":"ContainerStarted","Data":"c7713a2d6d781f7be0f4bdd8fd2872bc477417589fafb6ea014feb75c1655316"}
Sep 05 02:19:42.049335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:42.048997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b" event={"ID":"d9a4adad-8278-4253-a58e-a20c3f4464f6","Type":"ContainerStarted","Data":"9a17630f1f2ee57a54a49c0210f709bc9d4c21fabc78fba90bc3d2d845abc7fe"}
Sep 05 02:19:42.049335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:42.049017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b" event={"ID":"d9a4adad-8278-4253-a58e-a20c3f4464f6","Type":"ContainerStarted","Data":"def199ff8446aea53f0dc0de2967e488376282a83e8dac54c6b3b84b7f13f41e"}
Sep 05 02:19:43.055050 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:43.055012    2579 generic.go:356] "Generic (PLEG): container finished" podID="5d817548-f439-4d19-b3c5-c74c8a1c92f8" containerID="ba1ac328a1968a71c1b020835e33e7ae8604d1af8b8035d536948c8c588d710e" exitCode=0
Sep 05 02:19:43.055453 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:43.055089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x57pm" event={"ID":"5d817548-f439-4d19-b3c5-c74c8a1c92f8","Type":"ContainerDied","Data":"ba1ac328a1968a71c1b020835e33e7ae8604d1af8b8035d536948c8c588d710e"}
Sep 05 02:19:43.057314 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:43.057286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct" event={"ID":"30d67f0e-58fc-46f2-8e09-d68e9a20d747","Type":"ContainerStarted","Data":"7b1db9c582827194b25fbe4d767b17abc75cc73641870c67fb29f943bc51f1d4"}
Sep 05 02:19:43.057402 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:43.057345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct" event={"ID":"30d67f0e-58fc-46f2-8e09-d68e9a20d747","Type":"ContainerStarted","Data":"4b2bb7eec4b3bdb0f21ac666d4bb9b6e647da85fa9bb6d768f13cfad8ced1f9f"}
Sep 05 02:19:44.063192 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.063154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x57pm" event={"ID":"5d817548-f439-4d19-b3c5-c74c8a1c92f8","Type":"ContainerStarted","Data":"3bbd41ed8701b5ca4ef7978e84370ee9e0ccbd88c329f1fc84e1a5c03d98fa16"}
Sep 05 02:19:44.063745 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.063207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x57pm" event={"ID":"5d817548-f439-4d19-b3c5-c74c8a1c92f8","Type":"ContainerStarted","Data":"260f9befc3a9ed3ed9b966a50872dbebaa4a29aa9c3f0b3df10b1821b0c693aa"}
Sep 05 02:19:44.065469 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.065434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b" event={"ID":"d9a4adad-8278-4253-a58e-a20c3f4464f6","Type":"ContainerStarted","Data":"98d79a9edfad2426ce7841c85e2b1158bbb0c374e43bbb06d2a349baa78704b2"}
Sep 05 02:19:44.067453 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.067427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct" event={"ID":"30d67f0e-58fc-46f2-8e09-d68e9a20d747","Type":"ContainerStarted","Data":"b90a516c745c39ba3a5aeea22354d15e6d7b1c74b2a9c96b8b76cf28c61a7d85"}
Sep 05 02:19:44.084409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.084367    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x57pm" podStartSLOduration=3.284468735 podStartE2EDuration="5.08435155s" totalImagesPullingTime="1.69987306s" totalInitContainerRuntime="100.009755ms" isStatefulPod=true podCreationTimestamp="2026-09-05 02:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:44.083479305 +0000 UTC m=+128.055783181" watchObservedRunningTime="2026-09-05 02:19:44.08435155 +0000 UTC m=+128.056655408"
Sep 05 02:19:44.104732 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.104628    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-242ct" podStartSLOduration=3.115598085 podStartE2EDuration="5.104608308s" totalImagesPullingTime="1.989010223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:44.10396643 +0000 UTC m=+128.076270307" watchObservedRunningTime="2026-09-05 02:19:44.104608308 +0000 UTC m=+128.076912185"
Sep 05 02:19:44.123069 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.122996    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-w6f6b" podStartSLOduration=3.144964691 podStartE2EDuration="5.122965945s" totalImagesPullingTime="1.978001254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:44.121554927 +0000 UTC m=+128.093858804" watchObservedRunningTime="2026-09-05 02:19:44.122965945 +0000 UTC m=+128.095269822"
Sep 05 02:19:44.525362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.525325    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"]
Sep 05 02:19:44.542846 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.542823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:44.542998 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.542828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"]
Sep 05 02:19:44.545500 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.545470    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 05 02:19:44.546218 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.546198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-swpbp\""
Sep 05 02:19:44.608514 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.608486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d27d2fa-7c74-4d65-8554-56b9909d589c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-lztgm\" (UID: \"9d27d2fa-7c74-4d65-8554-56b9909d589c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:44.608645 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.608541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkn4n\" (UniqueName: \"kubernetes.io/projected/9d27d2fa-7c74-4d65-8554-56b9909d589c-kube-api-access-fkn4n\") pod \"monitoring-plugin-85bbbbbf4-lztgm\" (UID: \"9d27d2fa-7c74-4d65-8554-56b9909d589c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:44.709858 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.709816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkn4n\" (UniqueName: \"kubernetes.io/projected/9d27d2fa-7c74-4d65-8554-56b9909d589c-kube-api-access-fkn4n\") pod \"monitoring-plugin-85bbbbbf4-lztgm\" (UID: \"9d27d2fa-7c74-4d65-8554-56b9909d589c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:44.710006 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.709908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d27d2fa-7c74-4d65-8554-56b9909d589c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-lztgm\" (UID: \"9d27d2fa-7c74-4d65-8554-56b9909d589c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:44.710006 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:44.710001    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 05 02:19:44.710087 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:19:44.710052    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d27d2fa-7c74-4d65-8554-56b9909d589c-monitoring-plugin-cert podName:9d27d2fa-7c74-4d65-8554-56b9909d589c nodeName:}" failed. No retries permitted until 2026-09-05 02:19:45.210038004 +0000 UTC m=+129.182341857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9d27d2fa-7c74-4d65-8554-56b9909d589c-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-lztgm" (UID: "9d27d2fa-7c74-4d65-8554-56b9909d589c") : secret "monitoring-plugin-cert" not found
Sep 05 02:19:44.723770 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:44.723741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkn4n\" (UniqueName: \"kubernetes.io/projected/9d27d2fa-7c74-4d65-8554-56b9909d589c-kube-api-access-fkn4n\") pod \"monitoring-plugin-85bbbbbf4-lztgm\" (UID: \"9d27d2fa-7c74-4d65-8554-56b9909d589c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:45.215507 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.215400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d27d2fa-7c74-4d65-8554-56b9909d589c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-lztgm\" (UID: \"9d27d2fa-7c74-4d65-8554-56b9909d589c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:45.220523 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.220466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d27d2fa-7c74-4d65-8554-56b9909d589c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-lztgm\" (UID: \"9d27d2fa-7c74-4d65-8554-56b9909d589c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:45.453617 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.453580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:45.524874 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.524764    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86fb44966-ttlhr"]
Sep 05 02:19:45.549620 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.549565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86fb44966-ttlhr"]
Sep 05 02:19:45.549818 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.549679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.620584 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.620551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxzd\" (UniqueName: \"kubernetes.io/projected/0832883f-f3fc-4231-9a0b-dae2dd49927f-kube-api-access-nmxzd\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.620754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.620589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-service-ca\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.620754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.620646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-trusted-ca-bundle\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.620754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.620708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-serving-cert\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.620754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.620729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-oauth-config\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.620897 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.620770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-config\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.620897 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.620804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-oauth-serving-cert\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.721571 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.721534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-service-ca\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.721779 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.721591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-trusted-ca-bundle\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.721779 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.721637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-serving-cert\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.721779 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.721689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-oauth-config\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.721779 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.721751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-config\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.722005 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.721782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-oauth-serving-cert\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.722005 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.721848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxzd\" (UniqueName: \"kubernetes.io/projected/0832883f-f3fc-4231-9a0b-dae2dd49927f-kube-api-access-nmxzd\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.722320 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.722293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-service-ca\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.722654 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.722629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-trusted-ca-bundle\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.722789 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.722715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-oauth-serving-cert\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.722986 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.722965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-config\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.724646 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.724623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-serving-cert\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.724872 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.724848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-oauth-config\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.735025 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.734969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxzd\" (UniqueName: \"kubernetes.io/projected/0832883f-f3fc-4231-9a0b-dae2dd49927f-kube-api-access-nmxzd\") pod \"console-86fb44966-ttlhr\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") " pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:45.759732 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:45.759703    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d27d2fa_7c74_4d65_8554_56b9909d589c.slice/crio-594ead753d9091c533945afb08e79ea23c26f3b37f816a45a6f12fdb7e4b096d WatchSource:0}: Error finding container 594ead753d9091c533945afb08e79ea23c26f3b37f816a45a6f12fdb7e4b096d: Status 404 returned error can't find the container with id 594ead753d9091c533945afb08e79ea23c26f3b37f816a45a6f12fdb7e4b096d
Sep 05 02:19:45.783914 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.783886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"]
Sep 05 02:19:45.865417 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:45.865378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:46.076195 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:46.076157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm" event={"ID":"9d27d2fa-7c74-4d65-8554-56b9909d589c","Type":"ContainerStarted","Data":"594ead753d9091c533945afb08e79ea23c26f3b37f816a45a6f12fdb7e4b096d"}
Sep 05 02:19:46.185950 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:46.185914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86fb44966-ttlhr"]
Sep 05 02:19:46.188766 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:46.188737    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0832883f_f3fc_4231_9a0b_dae2dd49927f.slice/crio-2d45b52c22eda339a352f820f94f79dc0cfb97a315ce4cf3cd2f5e9aa81536d1 WatchSource:0}: Error finding container 2d45b52c22eda339a352f820f94f79dc0cfb97a315ce4cf3cd2f5e9aa81536d1: Status 404 returned error can't find the container with id 2d45b52c22eda339a352f820f94f79dc0cfb97a315ce4cf3cd2f5e9aa81536d1
Sep 05 02:19:46.226043 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:46.226008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:19:46.228458 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:46.228430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4662252d-cd26-4d5c-ad62-0c815a71e462-metrics-certs\") pod \"network-metrics-daemon-xk94l\" (UID: \"4662252d-cd26-4d5c-ad62-0c815a71e462\") " pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:19:46.365156 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:46.365079    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-66tlm\""
Sep 05 02:19:46.373266 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:46.373235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk94l"
Sep 05 02:19:46.658531 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:46.658134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk94l"]
Sep 05 02:19:46.660785 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:19:46.660755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4662252d_cd26_4d5c_ad62_0c815a71e462.slice/crio-f888829c712c9df16adb4f289d1f5460475e5e34d51ca3c4ca00495fa8aa2283 WatchSource:0}: Error finding container f888829c712c9df16adb4f289d1f5460475e5e34d51ca3c4ca00495fa8aa2283: Status 404 returned error can't find the container with id f888829c712c9df16adb4f289d1f5460475e5e34d51ca3c4ca00495fa8aa2283
Sep 05 02:19:47.081335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:47.081299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk94l" event={"ID":"4662252d-cd26-4d5c-ad62-0c815a71e462","Type":"ContainerStarted","Data":"f888829c712c9df16adb4f289d1f5460475e5e34d51ca3c4ca00495fa8aa2283"}
Sep 05 02:19:47.082776 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:47.082751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86fb44966-ttlhr" event={"ID":"0832883f-f3fc-4231-9a0b-dae2dd49927f","Type":"ContainerStarted","Data":"31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9"}
Sep 05 02:19:47.082887 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:47.082787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86fb44966-ttlhr" event={"ID":"0832883f-f3fc-4231-9a0b-dae2dd49927f","Type":"ContainerStarted","Data":"2d45b52c22eda339a352f820f94f79dc0cfb97a315ce4cf3cd2f5e9aa81536d1"}
Sep 05 02:19:47.103071 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:47.103029    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86fb44966-ttlhr" podStartSLOduration=2.103012564 podStartE2EDuration="2.103012564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:47.101622289 +0000 UTC m=+131.073926166" watchObservedRunningTime="2026-09-05 02:19:47.103012564 +0000 UTC m=+131.075316521"
Sep 05 02:19:49.093462 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:49.093412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk94l" event={"ID":"4662252d-cd26-4d5c-ad62-0c815a71e462","Type":"ContainerStarted","Data":"4303599433a93a64afc4cd36a87e408ce403c0c2bb33c8ff56de18b58dafefa5"}
Sep 05 02:19:49.093887 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:49.093490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk94l" event={"ID":"4662252d-cd26-4d5c-ad62-0c815a71e462","Type":"ContainerStarted","Data":"14c3f095a3759bdbc40823406b64925383d2f496ecad77e50e4d05b89199007c"}
Sep 05 02:19:49.112800 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:49.112749    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xk94l" podStartSLOduration=130.060796559 podStartE2EDuration="2m12.112733028s" totalImagesPullingTime="2.051936469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:49.112333161 +0000 UTC m=+133.084637040" watchObservedRunningTime="2026-09-05 02:19:49.112733028 +0000 UTC m=+133.085036932"
Sep 05 02:19:50.098621 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:50.098577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm" event={"ID":"9d27d2fa-7c74-4d65-8554-56b9909d589c","Type":"ContainerStarted","Data":"5b2969a2186e040af74207eefa1d201cb5f9e9fa6f609e18ce0699eaec94a307"}
Sep 05 02:19:50.099087 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:50.098817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:50.103436 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:50.103413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm"
Sep 05 02:19:50.114876 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:50.114825    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-lztgm" podStartSLOduration=2.77764234 podStartE2EDuration="6.114807026s" totalImagesPullingTime="3.337164686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:19:50.114630605 +0000 UTC m=+134.086934483" watchObservedRunningTime="2026-09-05 02:19:50.114807026 +0000 UTC m=+134.087110903"
Sep 05 02:19:55.865927 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:55.865883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:55.866401 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:55.865948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:55.870792 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:55.870767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:56.127001 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:56.126917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:19:56.179771 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:19:56.179734    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f67c845cd-tdvwv"]
Sep 05 02:20:01.692833 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:01.692765    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68" podUID="e67d164f-d7b1-4a02-b7ae-c09ccc990765" containerName="registry" containerID="cri-o://4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c" gracePeriod=30
Sep 05 02:20:01.933288 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:01.933262    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:20:02.060726 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.060684    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-installation-pull-secrets\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.060936 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.060740    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8nn4\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-kube-api-access-g8nn4\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.060936 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.060766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e67d164f-d7b1-4a02-b7ae-c09ccc990765-ca-trust-extracted\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.060936 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.060802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.060936 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.060898    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-certificates\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.061170 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.061004    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-image-registry-private-configuration\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.061170 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.061055    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-bound-sa-token\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.061170 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.061093    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-trusted-ca\") pod \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\" (UID: \"e67d164f-d7b1-4a02-b7ae-c09ccc990765\") "
Sep 05 02:20:02.061731 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.061649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:02.061852 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.061782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:02.063330 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.063300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:02.063514 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.063495    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-kube-api-access-g8nn4" (OuterVolumeSpecName: "kube-api-access-g8nn4") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "kube-api-access-g8nn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:02.063581 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.063518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:02.063707 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.063687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:02.063779 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.063704    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:02.069311 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.069286    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67d164f-d7b1-4a02-b7ae-c09ccc990765-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e67d164f-d7b1-4a02-b7ae-c09ccc990765" (UID: "e67d164f-d7b1-4a02-b7ae-c09ccc990765"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:20:02.145743 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.145703    2579 generic.go:356] "Generic (PLEG): container finished" podID="e67d164f-d7b1-4a02-b7ae-c09ccc990765" containerID="4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c" exitCode=0
Sep 05 02:20:02.145902 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.145747    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68"
Sep 05 02:20:02.145902 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.145791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68" event={"ID":"e67d164f-d7b1-4a02-b7ae-c09ccc990765","Type":"ContainerDied","Data":"4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c"}
Sep 05 02:20:02.145902 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.145842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57c954cd9c-kbj68" event={"ID":"e67d164f-d7b1-4a02-b7ae-c09ccc990765","Type":"ContainerDied","Data":"317bb4eb9cf3c18b1dc38ec9ba71fd88c7b53797ec3ec2e878fb1291cd714469"}
Sep 05 02:20:02.145902 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.145865    2579 scope.go:117] "RemoveContainer" containerID="4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c"
Sep 05 02:20:02.162201 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162176    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.162201 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162200    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-registry-certificates\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.162343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162210    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-image-registry-private-configuration\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.162343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162219    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-bound-sa-token\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.162343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162227    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e67d164f-d7b1-4a02-b7ae-c09ccc990765-trusted-ca\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.162343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162235    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e67d164f-d7b1-4a02-b7ae-c09ccc990765-installation-pull-secrets\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.162343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162242    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8nn4\" (UniqueName: \"kubernetes.io/projected/e67d164f-d7b1-4a02-b7ae-c09ccc990765-kube-api-access-g8nn4\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.162343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.162250    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e67d164f-d7b1-4a02-b7ae-c09ccc990765-ca-trust-extracted\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:02.167557 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.167482    2579 scope.go:117] "RemoveContainer" containerID="4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c"
Sep 05 02:20:02.168570 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:20:02.168054    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c\": container with ID starting with 4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c not found: ID does not exist" containerID="4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c"
Sep 05 02:20:02.168570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.168100    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c"} err="failed to get container status \"4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c\": rpc error: code = NotFound desc = could not find container \"4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c\": container with ID starting with 4aae9be40cbcc4063990b7e49aa6fbbfe90a8b7a38ce8434f692ef5c5a23823c not found: ID does not exist"
Sep 05 02:20:02.169830 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.169809    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57c954cd9c-kbj68"]
Sep 05 02:20:02.173528 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.173502    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57c954cd9c-kbj68"]
Sep 05 02:20:02.550683 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:02.550629    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e67d164f-d7b1-4a02-b7ae-c09ccc990765" path="/var/lib/kubelet/pods/e67d164f-d7b1-4a02-b7ae-c09ccc990765/volumes"
Sep 05 02:20:05.144727 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.144689    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dccf85fb9-2x9mr" podUID="dfe2e105-5f18-4566-958d-006a467010a0" containerName="console" containerID="cri-o://0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f" gracePeriod=15
Sep 05 02:20:05.429215 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.429185    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:20:05.592874 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.592836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-service-ca\") pod \"dfe2e105-5f18-4566-958d-006a467010a0\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") "
Sep 05 02:20:05.592874 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.592884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-serving-cert\") pod \"dfe2e105-5f18-4566-958d-006a467010a0\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") "
Sep 05 02:20:05.593119 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.592908    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-oauth-serving-cert\") pod \"dfe2e105-5f18-4566-958d-006a467010a0\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") "
Sep 05 02:20:05.593119 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.592989    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvnwl\" (UniqueName: \"kubernetes.io/projected/dfe2e105-5f18-4566-958d-006a467010a0-kube-api-access-mvnwl\") pod \"dfe2e105-5f18-4566-958d-006a467010a0\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") "
Sep 05 02:20:05.593119 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.593023    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-console-config\") pod \"dfe2e105-5f18-4566-958d-006a467010a0\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") "
Sep 05 02:20:05.593119 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.593051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-oauth-config\") pod \"dfe2e105-5f18-4566-958d-006a467010a0\" (UID: \"dfe2e105-5f18-4566-958d-006a467010a0\") "
Sep 05 02:20:05.593518 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.593342    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "dfe2e105-5f18-4566-958d-006a467010a0" (UID: "dfe2e105-5f18-4566-958d-006a467010a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:05.593518 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.593461    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dfe2e105-5f18-4566-958d-006a467010a0" (UID: "dfe2e105-5f18-4566-958d-006a467010a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:05.593518 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.593485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-console-config" (OuterVolumeSpecName: "console-config") pod "dfe2e105-5f18-4566-958d-006a467010a0" (UID: "dfe2e105-5f18-4566-958d-006a467010a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:05.595250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.595229    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dfe2e105-5f18-4566-958d-006a467010a0" (UID: "dfe2e105-5f18-4566-958d-006a467010a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:05.595310 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.595294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe2e105-5f18-4566-958d-006a467010a0-kube-api-access-mvnwl" (OuterVolumeSpecName: "kube-api-access-mvnwl") pod "dfe2e105-5f18-4566-958d-006a467010a0" (UID: "dfe2e105-5f18-4566-958d-006a467010a0"). InnerVolumeSpecName "kube-api-access-mvnwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:05.595354 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.595299    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dfe2e105-5f18-4566-958d-006a467010a0" (UID: "dfe2e105-5f18-4566-958d-006a467010a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:05.694480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.694397    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvnwl\" (UniqueName: \"kubernetes.io/projected/dfe2e105-5f18-4566-958d-006a467010a0-kube-api-access-mvnwl\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:05.694480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.694428    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-console-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:05.694480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.694440    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-oauth-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:05.694480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.694450    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-service-ca\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:05.694480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.694458    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dfe2e105-5f18-4566-958d-006a467010a0-console-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:05.694480 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:05.694467    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dfe2e105-5f18-4566-958d-006a467010a0-oauth-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:06.163776 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.163742    2579 generic.go:356] "Generic (PLEG): container finished" podID="dfe2e105-5f18-4566-958d-006a467010a0" containerID="0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f" exitCode=0
Sep 05 02:20:06.164194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.163789    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccf85fb9-2x9mr"
Sep 05 02:20:06.164194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.163788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccf85fb9-2x9mr" event={"ID":"dfe2e105-5f18-4566-958d-006a467010a0","Type":"ContainerDied","Data":"0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f"}
Sep 05 02:20:06.164194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.163923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccf85fb9-2x9mr" event={"ID":"dfe2e105-5f18-4566-958d-006a467010a0","Type":"ContainerDied","Data":"620a6e0d791357617f7055aa83572fac0e3af1c308452d73623ce0850caf8c89"}
Sep 05 02:20:06.164194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.163947    2579 scope.go:117] "RemoveContainer" containerID="0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f"
Sep 05 02:20:06.172063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.172040    2579 scope.go:117] "RemoveContainer" containerID="0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f"
Sep 05 02:20:06.172316 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:20:06.172297    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f\": container with ID starting with 0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f not found: ID does not exist" containerID="0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f"
Sep 05 02:20:06.172382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.172326    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f"} err="failed to get container status \"0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f\": rpc error: code = NotFound desc = could not find container \"0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f\": container with ID starting with 0e59cf5c95df73b580fbea0425e65f7887f0e680fbae6cf10a1da40b7f6f0e3f not found: ID does not exist"
Sep 05 02:20:06.190600 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.187138    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dccf85fb9-2x9mr"]
Sep 05 02:20:06.192853 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.192828    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dccf85fb9-2x9mr"]
Sep 05 02:20:06.550723 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:06.550690    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfe2e105-5f18-4566-958d-006a467010a0" path="/var/lib/kubelet/pods/dfe2e105-5f18-4566-958d-006a467010a0/volumes"
Sep 05 02:20:16.204972 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:16.204931    2579 generic.go:356] "Generic (PLEG): container finished" podID="57df1726-1a5b-4ca3-828f-3c39ccbb533c" containerID="52cda29427527e946ccca07a0cd83bdd7fd6c9484a01fac871b35d0329c7bee6" exitCode=0
Sep 05 02:20:16.205383 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:16.205001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f" event={"ID":"57df1726-1a5b-4ca3-828f-3c39ccbb533c","Type":"ContainerDied","Data":"52cda29427527e946ccca07a0cd83bdd7fd6c9484a01fac871b35d0329c7bee6"}
Sep 05 02:20:16.205383 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:16.205348    2579 scope.go:117] "RemoveContainer" containerID="52cda29427527e946ccca07a0cd83bdd7fd6c9484a01fac871b35d0329c7bee6"
Sep 05 02:20:17.212483 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:17.212446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-x7g2f" event={"ID":"57df1726-1a5b-4ca3-828f-3c39ccbb533c","Type":"ContainerStarted","Data":"108345fe4c77b0cac566fce94fc77e566107455c6964aba153ab2fc9f99f47b0"}
Sep 05 02:20:21.200103 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.200054    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f67c845cd-tdvwv" podUID="ca66ac21-f7f0-45ce-a1a0-296e35f2020a" containerName="console" containerID="cri-o://f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18" gracePeriod=15
Sep 05 02:20:21.469722 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.469637    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:20:21.511152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-oauth-serving-cert\") pod \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") "
Sep 05 02:20:21.511326 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511187    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-trusted-ca-bundle\") pod \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") "
Sep 05 02:20:21.511326 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-serving-cert\") pod \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") "
Sep 05 02:20:21.511326 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-oauth-config\") pod \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") "
Sep 05 02:20:21.511488 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-service-ca\") pod \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") "
Sep 05 02:20:21.511488 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-config\") pod \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") "
Sep 05 02:20:21.511488 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfcdt\" (UniqueName: \"kubernetes.io/projected/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-kube-api-access-qfcdt\") pod \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\" (UID: \"ca66ac21-f7f0-45ce-a1a0-296e35f2020a\") "
Sep 05 02:20:21.511634 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511558    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ca66ac21-f7f0-45ce-a1a0-296e35f2020a" (UID: "ca66ac21-f7f0-45ce-a1a0-296e35f2020a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:21.511745 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511728    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-oauth-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:21.511745 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ca66ac21-f7f0-45ce-a1a0-296e35f2020a" (UID: "ca66ac21-f7f0-45ce-a1a0-296e35f2020a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:21.511878 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-service-ca" (OuterVolumeSpecName: "service-ca") pod "ca66ac21-f7f0-45ce-a1a0-296e35f2020a" (UID: "ca66ac21-f7f0-45ce-a1a0-296e35f2020a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:21.512033 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.511984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-config" (OuterVolumeSpecName: "console-config") pod "ca66ac21-f7f0-45ce-a1a0-296e35f2020a" (UID: "ca66ac21-f7f0-45ce-a1a0-296e35f2020a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:20:21.513358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.513338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ca66ac21-f7f0-45ce-a1a0-296e35f2020a" (UID: "ca66ac21-f7f0-45ce-a1a0-296e35f2020a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:21.513872 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.513841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-kube-api-access-qfcdt" (OuterVolumeSpecName: "kube-api-access-qfcdt") pod "ca66ac21-f7f0-45ce-a1a0-296e35f2020a" (UID: "ca66ac21-f7f0-45ce-a1a0-296e35f2020a"). InnerVolumeSpecName "kube-api-access-qfcdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:20:21.513959 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.513940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ca66ac21-f7f0-45ce-a1a0-296e35f2020a" (UID: "ca66ac21-f7f0-45ce-a1a0-296e35f2020a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:20:21.613088 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.613042    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-trusted-ca-bundle\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:21.613088 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.613083    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:21.613088 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.613097    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-oauth-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:21.613376 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.613109    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-service-ca\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:21.613376 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.613119    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-console-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:21.613376 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:21.613128    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcdt\" (UniqueName: \"kubernetes.io/projected/ca66ac21-f7f0-45ce-a1a0-296e35f2020a-kube-api-access-qfcdt\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:20:22.231466 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.231433    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca66ac21-f7f0-45ce-a1a0-296e35f2020a" containerID="f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18" exitCode=0
Sep 05 02:20:22.231908 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.231480    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f67c845cd-tdvwv"
Sep 05 02:20:22.231908 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.231503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f67c845cd-tdvwv" event={"ID":"ca66ac21-f7f0-45ce-a1a0-296e35f2020a","Type":"ContainerDied","Data":"f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18"}
Sep 05 02:20:22.231908 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.231541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f67c845cd-tdvwv" event={"ID":"ca66ac21-f7f0-45ce-a1a0-296e35f2020a","Type":"ContainerDied","Data":"80ab1624913170ac1b803d0a75356d38e8d9bc0bfb1737bb147d55d5e48054b8"}
Sep 05 02:20:22.231908 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.231560    2579 scope.go:117] "RemoveContainer" containerID="f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18"
Sep 05 02:20:22.240174 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.240155    2579 scope.go:117] "RemoveContainer" containerID="f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18"
Sep 05 02:20:22.240472 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:20:22.240450    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18\": container with ID starting with f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18 not found: ID does not exist" containerID="f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18"
Sep 05 02:20:22.240519 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.240481    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18"} err="failed to get container status \"f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18\": rpc error: code = NotFound desc = could not find container \"f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18\": container with ID starting with f84cd213a40817dfb7841dce41704fdea7e348380cf3669f967c97f001a07d18 not found: ID does not exist"
Sep 05 02:20:22.253172 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.253141    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f67c845cd-tdvwv"]
Sep 05 02:20:22.258642 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.258620    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f67c845cd-tdvwv"]
Sep 05 02:20:22.549366 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:20:22.549328    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca66ac21-f7f0-45ce-a1a0-296e35f2020a" path="/var/lib/kubelet/pods/ca66ac21-f7f0-45ce-a1a0-296e35f2020a/volumes"
Sep 05 02:21:04.221227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221188    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"]
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221499    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67d164f-d7b1-4a02-b7ae-c09ccc990765" containerName="registry"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221510    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67d164f-d7b1-4a02-b7ae-c09ccc990765" containerName="registry"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221527    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe2e105-5f18-4566-958d-006a467010a0" containerName="console"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221542    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe2e105-5f18-4566-958d-006a467010a0" containerName="console"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca66ac21-f7f0-45ce-a1a0-296e35f2020a" containerName="console"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221558    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca66ac21-f7f0-45ce-a1a0-296e35f2020a" containerName="console"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221615    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67d164f-d7b1-4a02-b7ae-c09ccc990765" containerName="registry"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221650    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe2e105-5f18-4566-958d-006a467010a0" containerName="console"
Sep 05 02:21:04.221720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.221657    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca66ac21-f7f0-45ce-a1a0-296e35f2020a" containerName="console"
Sep 05 02:21:04.223997 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.223981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.228261 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.228238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 05 02:21:04.228408 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.228315    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 05 02:21:04.228495 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.228471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-f64vq\""
Sep 05 02:21:04.228676 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.228647    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 05 02:21:04.228759 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.228683    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 05 02:21:04.229824 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.229644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 05 02:21:04.233758 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.233738    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 05 02:21:04.238077 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.238057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"]
Sep 05 02:21:04.263226 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.263226 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-telemeter-client-tls\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.263439 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-serving-certs-ca-bundle\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.263439 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-metrics-client-ca\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.263439 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-secret-telemeter-client\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.263439 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.263625 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhdkt\" (UniqueName: \"kubernetes.io/projected/0c549329-240e-49d6-965c-aa1d755f5311-kube-api-access-mhdkt\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.263625 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.263520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-federate-client-tls\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364061 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdkt\" (UniqueName: \"kubernetes.io/projected/0c549329-240e-49d6-965c-aa1d755f5311-kube-api-access-mhdkt\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-federate-client-tls\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-telemeter-client-tls\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-serving-certs-ca-bundle\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-metrics-client-ca\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-secret-telemeter-client\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.364250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.364236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.365087 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.365056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-metrics-client-ca\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.365182 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.365154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-serving-certs-ca-bundle\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.365222 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.365192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c549329-240e-49d6-965c-aa1d755f5311-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.366968 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.366946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-secret-telemeter-client\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.367059 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.366987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-telemeter-client-tls\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.367275 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.367252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.367311 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.367286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0c549329-240e-49d6-965c-aa1d755f5311-federate-client-tls\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.371870 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.371854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdkt\" (UniqueName: \"kubernetes.io/projected/0c549329-240e-49d6-965c-aa1d755f5311-kube-api-access-mhdkt\") pod \"telemeter-client-5dd57d64b7-g76gj\" (UID: \"0c549329-240e-49d6-965c-aa1d755f5311\") " pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.533603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.533561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"
Sep 05 02:21:04.810578 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:04.810549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dd57d64b7-g76gj"]
Sep 05 02:21:04.812755 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:21:04.812718    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c549329_240e_49d6_965c_aa1d755f5311.slice/crio-e47d0f48f45d7e3497b62111dd6ef8e947b2db951c81858c87d4528e642447bb WatchSource:0}: Error finding container e47d0f48f45d7e3497b62111dd6ef8e947b2db951c81858c87d4528e642447bb: Status 404 returned error can't find the container with id e47d0f48f45d7e3497b62111dd6ef8e947b2db951c81858c87d4528e642447bb
Sep 05 02:21:05.409698 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:05.409650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj" event={"ID":"0c549329-240e-49d6-965c-aa1d755f5311","Type":"ContainerStarted","Data":"e47d0f48f45d7e3497b62111dd6ef8e947b2db951c81858c87d4528e642447bb"}
Sep 05 02:21:07.419426 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:07.419393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj" event={"ID":"0c549329-240e-49d6-965c-aa1d755f5311","Type":"ContainerStarted","Data":"cac3d8497d4f54e03d2631900905c4b3f3c30a4cd5c0869aceaee5f214484ca3"}
Sep 05 02:21:08.425597 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.425560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj" event={"ID":"0c549329-240e-49d6-965c-aa1d755f5311","Type":"ContainerStarted","Data":"d4c0e49b8de9036ea93eb26abc0e8a40f2b8607cb7fab8ab4b1d2aa6cb683dc9"}
Sep 05 02:21:08.426114 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.425634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj" event={"ID":"0c549329-240e-49d6-965c-aa1d755f5311","Type":"ContainerStarted","Data":"bbb76fe7c1a66725fe4a843bc2f48a740f7a436615a26ae0bd42a752e1b32463"}
Sep 05 02:21:08.450055 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.450000    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dd57d64b7-g76gj" podStartSLOduration=1.829043526 podStartE2EDuration="4.449984214s" totalImagesPullingTime="2.620940688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:21:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:08.449374601 +0000 UTC m=+212.421678511" watchObservedRunningTime="2026-09-05 02:21:08.449984214 +0000 UTC m=+212.422288093"
Sep 05 02:21:08.533927 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.533892    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b5cb5f55b-nxtbd"]
Sep 05 02:21:08.536193 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.536174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.550455 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.550429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5cb5f55b-nxtbd"]
Sep 05 02:21:08.593063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.593027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-serving-cert\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.593063 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.593067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-trusted-ca-bundle\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.593308 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.593095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-console-config\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.593308 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.593184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-service-ca\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.593308 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.593227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzt7\" (UniqueName: \"kubernetes.io/projected/64e5870d-00fe-405f-b097-9195e38d45a1-kube-api-access-7lzt7\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.593308 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.593272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-oauth-config\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.593308 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.593300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-oauth-serving-cert\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.693997 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.693911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-oauth-config\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.693997 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.693953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-oauth-serving-cert\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.693997 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.693974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-serving-cert\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.694265 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-trusted-ca-bundle\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.694265 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-console-config\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.694265 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-service-ca\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.694265 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzt7\" (UniqueName: \"kubernetes.io/projected/64e5870d-00fe-405f-b097-9195e38d45a1-kube-api-access-7lzt7\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.694720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694688    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-oauth-serving-cert\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.694942 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-console-config\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.695013 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-service-ca\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.695013 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.694960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-trusted-ca-bundle\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.696479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.696452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-serving-cert\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.696570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.696547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-oauth-config\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.702866 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.702842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzt7\" (UniqueName: \"kubernetes.io/projected/64e5870d-00fe-405f-b097-9195e38d45a1-kube-api-access-7lzt7\") pod \"console-6b5cb5f55b-nxtbd\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") " pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:08.845576 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:08.845541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:09.115817 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:09.115649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5cb5f55b-nxtbd"]
Sep 05 02:21:09.118371 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:21:09.118333    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e5870d_00fe_405f_b097_9195e38d45a1.slice/crio-fc5d947fb888a57e0f9ce40ae1af1c72a2e290e2565e67a72b975ddb2c22c29c WatchSource:0}: Error finding container fc5d947fb888a57e0f9ce40ae1af1c72a2e290e2565e67a72b975ddb2c22c29c: Status 404 returned error can't find the container with id fc5d947fb888a57e0f9ce40ae1af1c72a2e290e2565e67a72b975ddb2c22c29c
Sep 05 02:21:09.430692 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:09.430567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5cb5f55b-nxtbd" event={"ID":"64e5870d-00fe-405f-b097-9195e38d45a1","Type":"ContainerStarted","Data":"7ed6e63653c3d3709a16c5c64717aa50aeeea4479df795b27639dce80f61cfd6"}
Sep 05 02:21:09.430692 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:09.430625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5cb5f55b-nxtbd" event={"ID":"64e5870d-00fe-405f-b097-9195e38d45a1","Type":"ContainerStarted","Data":"fc5d947fb888a57e0f9ce40ae1af1c72a2e290e2565e67a72b975ddb2c22c29c"}
Sep 05 02:21:09.449392 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:09.449345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b5cb5f55b-nxtbd" podStartSLOduration=1.449328087 podStartE2EDuration="1.449328087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:09.448134322 +0000 UTC m=+213.420438201" watchObservedRunningTime="2026-09-05 02:21:09.449328087 +0000 UTC m=+213.421632029"
Sep 05 02:21:17.407335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.407292    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5cb5f55b-nxtbd"]
Sep 05 02:21:17.451266 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.451231    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-656c55dc96-qlhhx"]
Sep 05 02:21:17.453936 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.453919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.464712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.464683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656c55dc96-qlhhx"]
Sep 05 02:21:17.564896 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.564859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-serving-cert\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.565031 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.564907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-trusted-ca-bundle\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.565031 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.564973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-oauth-config\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.565031 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.565002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68xf\" (UniqueName: \"kubernetes.io/projected/cf7d1add-d204-4141-a325-c32d7678414c-kube-api-access-d68xf\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.565150 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.565049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-console-config\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.565150 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.565134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-service-ca\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.565211 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.565175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-oauth-serving-cert\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.665853 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.665766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-service-ca\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.665853 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.665811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-oauth-serving-cert\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.665853 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.665832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-serving-cert\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.665853 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.665849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-trusted-ca-bundle\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.666201 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.665877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-oauth-config\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.666201 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.665990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68xf\" (UniqueName: \"kubernetes.io/projected/cf7d1add-d204-4141-a325-c32d7678414c-kube-api-access-d68xf\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.666201 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.666053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-console-config\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.666612 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.666585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-service-ca\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.666753 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.666682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-oauth-serving-cert\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.666798 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.666757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-trusted-ca-bundle\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.666884 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.666863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-console-config\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.668489 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.668464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-oauth-config\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.668570 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.668479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-serving-cert\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.675045 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.675023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68xf\" (UniqueName: \"kubernetes.io/projected/cf7d1add-d204-4141-a325-c32d7678414c-kube-api-access-d68xf\") pod \"console-656c55dc96-qlhhx\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") " pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:17.764589 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:17.764544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:18.034643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:18.034584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656c55dc96-qlhhx"]
Sep 05 02:21:18.037290 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:21:18.037261    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7d1add_d204_4141_a325_c32d7678414c.slice/crio-084dd2533599544973f6f224d4aaef51a07dd86cd41ded45ff95a40e5ef5bc5e WatchSource:0}: Error finding container 084dd2533599544973f6f224d4aaef51a07dd86cd41ded45ff95a40e5ef5bc5e: Status 404 returned error can't find the container with id 084dd2533599544973f6f224d4aaef51a07dd86cd41ded45ff95a40e5ef5bc5e
Sep 05 02:21:18.465619 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:18.465518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656c55dc96-qlhhx" event={"ID":"cf7d1add-d204-4141-a325-c32d7678414c","Type":"ContainerStarted","Data":"fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad"}
Sep 05 02:21:18.465619 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:18.465576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656c55dc96-qlhhx" event={"ID":"cf7d1add-d204-4141-a325-c32d7678414c","Type":"ContainerStarted","Data":"084dd2533599544973f6f224d4aaef51a07dd86cd41ded45ff95a40e5ef5bc5e"}
Sep 05 02:21:18.486345 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:18.486292    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-656c55dc96-qlhhx" podStartSLOduration=1.486275824 podStartE2EDuration="1.486275824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:21:18.48455873 +0000 UTC m=+222.456862606" watchObservedRunningTime="2026-09-05 02:21:18.486275824 +0000 UTC m=+222.458579699"
Sep 05 02:21:18.846683 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:18.846635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:27.765120 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:27.765084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:27.765597 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:27.765243    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:27.770090 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:27.770067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:28.507448 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:28.507418    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:21:28.556047 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:28.556014    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86fb44966-ttlhr"]
Sep 05 02:21:42.426755 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.426689    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b5cb5f55b-nxtbd" podUID="64e5870d-00fe-405f-b097-9195e38d45a1" containerName="console" containerID="cri-o://7ed6e63653c3d3709a16c5c64717aa50aeeea4479df795b27639dce80f61cfd6" gracePeriod=15
Sep 05 02:21:42.558571 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.558524    2579 generic.go:356] "Generic (PLEG): container finished" podID="64e5870d-00fe-405f-b097-9195e38d45a1" containerID="7ed6e63653c3d3709a16c5c64717aa50aeeea4479df795b27639dce80f61cfd6" exitCode=0
Sep 05 02:21:42.558743 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.558605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5cb5f55b-nxtbd" event={"ID":"64e5870d-00fe-405f-b097-9195e38d45a1","Type":"ContainerDied","Data":"7ed6e63653c3d3709a16c5c64717aa50aeeea4479df795b27639dce80f61cfd6"}
Sep 05 02:21:42.664913 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.664885    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:42.792072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-service-ca\") pod \"64e5870d-00fe-405f-b097-9195e38d45a1\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") "
Sep 05 02:21:42.792227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-oauth-serving-cert\") pod \"64e5870d-00fe-405f-b097-9195e38d45a1\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") "
Sep 05 02:21:42.792227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-console-config\") pod \"64e5870d-00fe-405f-b097-9195e38d45a1\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") "
Sep 05 02:21:42.792227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-serving-cert\") pod \"64e5870d-00fe-405f-b097-9195e38d45a1\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") "
Sep 05 02:21:42.792227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792173    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-trusted-ca-bundle\") pod \"64e5870d-00fe-405f-b097-9195e38d45a1\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") "
Sep 05 02:21:42.792227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lzt7\" (UniqueName: \"kubernetes.io/projected/64e5870d-00fe-405f-b097-9195e38d45a1-kube-api-access-7lzt7\") pod \"64e5870d-00fe-405f-b097-9195e38d45a1\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") "
Sep 05 02:21:42.792475 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-oauth-config\") pod \"64e5870d-00fe-405f-b097-9195e38d45a1\" (UID: \"64e5870d-00fe-405f-b097-9195e38d45a1\") "
Sep 05 02:21:42.792534 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-service-ca" (OuterVolumeSpecName: "service-ca") pod "64e5870d-00fe-405f-b097-9195e38d45a1" (UID: "64e5870d-00fe-405f-b097-9195e38d45a1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:42.792604 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792574    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-console-config" (OuterVolumeSpecName: "console-config") pod "64e5870d-00fe-405f-b097-9195e38d45a1" (UID: "64e5870d-00fe-405f-b097-9195e38d45a1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:42.792685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "64e5870d-00fe-405f-b097-9195e38d45a1" (UID: "64e5870d-00fe-405f-b097-9195e38d45a1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:42.792755 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.792717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "64e5870d-00fe-405f-b097-9195e38d45a1" (UID: "64e5870d-00fe-405f-b097-9195e38d45a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:42.794335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.794310    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e5870d-00fe-405f-b097-9195e38d45a1-kube-api-access-7lzt7" (OuterVolumeSpecName: "kube-api-access-7lzt7") pod "64e5870d-00fe-405f-b097-9195e38d45a1" (UID: "64e5870d-00fe-405f-b097-9195e38d45a1"). InnerVolumeSpecName "kube-api-access-7lzt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:42.794435 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.794334    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "64e5870d-00fe-405f-b097-9195e38d45a1" (UID: "64e5870d-00fe-405f-b097-9195e38d45a1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:42.794435 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.794385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "64e5870d-00fe-405f-b097-9195e38d45a1" (UID: "64e5870d-00fe-405f-b097-9195e38d45a1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:42.893193 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.893145    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzt7\" (UniqueName: \"kubernetes.io/projected/64e5870d-00fe-405f-b097-9195e38d45a1-kube-api-access-7lzt7\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:42.893193 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.893186    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-oauth-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:42.893193 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.893197    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-service-ca\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:42.893193 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.893206    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-oauth-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:42.893446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.893215    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-console-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:42.893446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.893223    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64e5870d-00fe-405f-b097-9195e38d45a1-console-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:42.893446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:42.893230    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e5870d-00fe-405f-b097-9195e38d45a1-trusted-ca-bundle\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:43.564978 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:43.564950    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5cb5f55b-nxtbd"
Sep 05 02:21:43.565377 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:43.564988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5cb5f55b-nxtbd" event={"ID":"64e5870d-00fe-405f-b097-9195e38d45a1","Type":"ContainerDied","Data":"fc5d947fb888a57e0f9ce40ae1af1c72a2e290e2565e67a72b975ddb2c22c29c"}
Sep 05 02:21:43.565377 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:43.565046    2579 scope.go:117] "RemoveContainer" containerID="7ed6e63653c3d3709a16c5c64717aa50aeeea4479df795b27639dce80f61cfd6"
Sep 05 02:21:43.585080 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:43.585051    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5cb5f55b-nxtbd"]
Sep 05 02:21:43.587284 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:43.587264    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b5cb5f55b-nxtbd"]
Sep 05 02:21:44.549632 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:44.549598    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e5870d-00fe-405f-b097-9195e38d45a1" path="/var/lib/kubelet/pods/64e5870d-00fe-405f-b097-9195e38d45a1/volumes"
Sep 05 02:21:53.578536 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.578467    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86fb44966-ttlhr" podUID="0832883f-f3fc-4231-9a0b-dae2dd49927f" containerName="console" containerID="cri-o://31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9" gracePeriod=15
Sep 05 02:21:53.814291 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.814267    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:21:53.987446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-oauth-serving-cert\") pod \"0832883f-f3fc-4231-9a0b-dae2dd49927f\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") "
Sep 05 02:21:53.987446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-serving-cert\") pod \"0832883f-f3fc-4231-9a0b-dae2dd49927f\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") "
Sep 05 02:21:53.987446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-trusted-ca-bundle\") pod \"0832883f-f3fc-4231-9a0b-dae2dd49927f\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") "
Sep 05 02:21:53.987755 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-config\") pod \"0832883f-f3fc-4231-9a0b-dae2dd49927f\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") "
Sep 05 02:21:53.987755 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987519    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-oauth-config\") pod \"0832883f-f3fc-4231-9a0b-dae2dd49927f\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") "
Sep 05 02:21:53.987755 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-service-ca\") pod \"0832883f-f3fc-4231-9a0b-dae2dd49927f\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") "
Sep 05 02:21:53.987755 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987601    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmxzd\" (UniqueName: \"kubernetes.io/projected/0832883f-f3fc-4231-9a0b-dae2dd49927f-kube-api-access-nmxzd\") pod \"0832883f-f3fc-4231-9a0b-dae2dd49927f\" (UID: \"0832883f-f3fc-4231-9a0b-dae2dd49927f\") "
Sep 05 02:21:53.988011 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987971    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0832883f-f3fc-4231-9a0b-dae2dd49927f" (UID: "0832883f-f3fc-4231-9a0b-dae2dd49927f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:53.988011 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0832883f-f3fc-4231-9a0b-dae2dd49927f" (UID: "0832883f-f3fc-4231-9a0b-dae2dd49927f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:53.988136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.987975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-config" (OuterVolumeSpecName: "console-config") pod "0832883f-f3fc-4231-9a0b-dae2dd49927f" (UID: "0832883f-f3fc-4231-9a0b-dae2dd49927f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:53.988136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.988095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-service-ca" (OuterVolumeSpecName: "service-ca") pod "0832883f-f3fc-4231-9a0b-dae2dd49927f" (UID: "0832883f-f3fc-4231-9a0b-dae2dd49927f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:21:53.989865 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.989830    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0832883f-f3fc-4231-9a0b-dae2dd49927f" (UID: "0832883f-f3fc-4231-9a0b-dae2dd49927f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:53.989865 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.989853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0832883f-f3fc-4231-9a0b-dae2dd49927f" (UID: "0832883f-f3fc-4231-9a0b-dae2dd49927f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:21:53.989996 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:53.989935    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0832883f-f3fc-4231-9a0b-dae2dd49927f-kube-api-access-nmxzd" (OuterVolumeSpecName: "kube-api-access-nmxzd") pod "0832883f-f3fc-4231-9a0b-dae2dd49927f" (UID: "0832883f-f3fc-4231-9a0b-dae2dd49927f"). InnerVolumeSpecName "kube-api-access-nmxzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:21:54.089194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.089159    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmxzd\" (UniqueName: \"kubernetes.io/projected/0832883f-f3fc-4231-9a0b-dae2dd49927f-kube-api-access-nmxzd\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:54.089194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.089185    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-oauth-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:54.089194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.089194    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:54.089194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.089203    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-trusted-ca-bundle\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:54.089441 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.089213    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:54.089441 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.089221    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0832883f-f3fc-4231-9a0b-dae2dd49927f-console-oauth-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:54.089441 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.089229    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0832883f-f3fc-4231-9a0b-dae2dd49927f-service-ca\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:21:54.605465 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.605432    2579 generic.go:356] "Generic (PLEG): container finished" podID="0832883f-f3fc-4231-9a0b-dae2dd49927f" containerID="31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9" exitCode=0
Sep 05 02:21:54.605888 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.605482    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86fb44966-ttlhr"
Sep 05 02:21:54.605888 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.605526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86fb44966-ttlhr" event={"ID":"0832883f-f3fc-4231-9a0b-dae2dd49927f","Type":"ContainerDied","Data":"31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9"}
Sep 05 02:21:54.605888 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.605577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86fb44966-ttlhr" event={"ID":"0832883f-f3fc-4231-9a0b-dae2dd49927f","Type":"ContainerDied","Data":"2d45b52c22eda339a352f820f94f79dc0cfb97a315ce4cf3cd2f5e9aa81536d1"}
Sep 05 02:21:54.605888 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.605597    2579 scope.go:117] "RemoveContainer" containerID="31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9"
Sep 05 02:21:54.613861 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.613831    2579 scope.go:117] "RemoveContainer" containerID="31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9"
Sep 05 02:21:54.614124 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:21:54.614098    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9\": container with ID starting with 31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9 not found: ID does not exist" containerID="31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9"
Sep 05 02:21:54.614173 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.614133    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9"} err="failed to get container status \"31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9\": rpc error: code = NotFound desc = could not find container \"31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9\": container with ID starting with 31d21b7d37ff486669863ae6d415c600f43ee2c1865c375757312b9b1be072f9 not found: ID does not exist"
Sep 05 02:21:54.620978 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.620952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86fb44966-ttlhr"]
Sep 05 02:21:54.625199 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:54.625180    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86fb44966-ttlhr"]
Sep 05 02:21:56.549876 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:21:56.549844    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0832883f-f3fc-4231-9a0b-dae2dd49927f" path="/var/lib/kubelet/pods/0832883f-f3fc-4231-9a0b-dae2dd49927f/volumes"
Sep 05 02:22:36.410454 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:22:36.410423    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 02:23:17.282409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.282373    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"]
Sep 05 02:23:17.282828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.282658    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e5870d-00fe-405f-b097-9195e38d45a1" containerName="console"
Sep 05 02:23:17.282828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.282683    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e5870d-00fe-405f-b097-9195e38d45a1" containerName="console"
Sep 05 02:23:17.282828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.282697    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0832883f-f3fc-4231-9a0b-dae2dd49927f" containerName="console"
Sep 05 02:23:17.282828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.282702    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0832883f-f3fc-4231-9a0b-dae2dd49927f" containerName="console"
Sep 05 02:23:17.282828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.282753    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0832883f-f3fc-4231-9a0b-dae2dd49927f" containerName="console"
Sep 05 02:23:17.282828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.282761    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e5870d-00fe-405f-b097-9195e38d45a1" containerName="console"
Sep 05 02:23:17.285638 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.285621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.288442 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.288421    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 02:23:17.288543 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.288421    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 02:23:17.289146 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.289128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hlckf\""
Sep 05 02:23:17.296879 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.296857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"]
Sep 05 02:23:17.323955 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.323931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.324098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.323981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7t5k\" (UniqueName: \"kubernetes.io/projected/7fd9ce54-76d8-499b-bb03-b0efe527fae4-kube-api-access-p7t5k\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.324098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.324003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.424338 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.424312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.424450 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.424379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7t5k\" (UniqueName: \"kubernetes.io/projected/7fd9ce54-76d8-499b-bb03-b0efe527fae4-kube-api-access-p7t5k\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.424450 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.424414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.424654 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.424624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.424752 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.424720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.432581 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.432560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7t5k\" (UniqueName: \"kubernetes.io/projected/7fd9ce54-76d8-499b-bb03-b0efe527fae4-kube-api-access-p7t5k\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.594343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.594291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:17.865087 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.865059    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"]
Sep 05 02:23:17.866609 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:23:17.866580    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd9ce54_76d8_499b_bb03_b0efe527fae4.slice/crio-7c61eb9ed79064d552b89f05722e9f8537c218792387e30ae7ecd0cb2278249a WatchSource:0}: Error finding container 7c61eb9ed79064d552b89f05722e9f8537c218792387e30ae7ecd0cb2278249a: Status 404 returned error can't find the container with id 7c61eb9ed79064d552b89f05722e9f8537c218792387e30ae7ecd0cb2278249a
Sep 05 02:23:17.868349 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.868332    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:23:17.922828 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:17.922800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng" event={"ID":"7fd9ce54-76d8-499b-bb03-b0efe527fae4","Type":"ContainerStarted","Data":"7c61eb9ed79064d552b89f05722e9f8537c218792387e30ae7ecd0cb2278249a"}
Sep 05 02:23:23.948276 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:23.948243    2579 generic.go:356] "Generic (PLEG): container finished" podID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerID="45a9206456bdc707e024d93bfafb024c16fb4c25834f6b1b85d47a048519887f" exitCode=0
Sep 05 02:23:23.948591 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:23.948295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng" event={"ID":"7fd9ce54-76d8-499b-bb03-b0efe527fae4","Type":"ContainerDied","Data":"45a9206456bdc707e024d93bfafb024c16fb4c25834f6b1b85d47a048519887f"}
Sep 05 02:23:26.961398 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:26.961361    2579 generic.go:356] "Generic (PLEG): container finished" podID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerID="97b6dcda26ee0a06454e1d2d1a0d2986e051522c141c305bf28607901c2ee37d" exitCode=0
Sep 05 02:23:26.961853 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:26.961407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng" event={"ID":"7fd9ce54-76d8-499b-bb03-b0efe527fae4","Type":"ContainerDied","Data":"97b6dcda26ee0a06454e1d2d1a0d2986e051522c141c305bf28607901c2ee37d"}
Sep 05 02:23:34.003768 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:34.003679    2579 generic.go:356] "Generic (PLEG): container finished" podID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerID="76172e23be9cb5b94bdd2b2fa5fbbe8df34e91d5dd9334e34eb9d41041ffe8ea" exitCode=0
Sep 05 02:23:34.003768 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:34.003727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng" event={"ID":"7fd9ce54-76d8-499b-bb03-b0efe527fae4","Type":"ContainerDied","Data":"76172e23be9cb5b94bdd2b2fa5fbbe8df34e91d5dd9334e34eb9d41041ffe8ea"}
Sep 05 02:23:35.128018 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.127995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:35.190890 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.190857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7t5k\" (UniqueName: \"kubernetes.io/projected/7fd9ce54-76d8-499b-bb03-b0efe527fae4-kube-api-access-p7t5k\") pod \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") "
Sep 05 02:23:35.191072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.190921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-bundle\") pod \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") "
Sep 05 02:23:35.191072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.190940    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-util\") pod \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\" (UID: \"7fd9ce54-76d8-499b-bb03-b0efe527fae4\") "
Sep 05 02:23:35.191613 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.191576    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-bundle" (OuterVolumeSpecName: "bundle") pod "7fd9ce54-76d8-499b-bb03-b0efe527fae4" (UID: "7fd9ce54-76d8-499b-bb03-b0efe527fae4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:23:35.193112 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.193090    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd9ce54-76d8-499b-bb03-b0efe527fae4-kube-api-access-p7t5k" (OuterVolumeSpecName: "kube-api-access-p7t5k") pod "7fd9ce54-76d8-499b-bb03-b0efe527fae4" (UID: "7fd9ce54-76d8-499b-bb03-b0efe527fae4"). InnerVolumeSpecName "kube-api-access-p7t5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:23:35.194957 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.194939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-util" (OuterVolumeSpecName: "util") pod "7fd9ce54-76d8-499b-bb03-b0efe527fae4" (UID: "7fd9ce54-76d8-499b-bb03-b0efe527fae4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:23:35.291623 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.291591    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7t5k\" (UniqueName: \"kubernetes.io/projected/7fd9ce54-76d8-499b-bb03-b0efe527fae4-kube-api-access-p7t5k\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:23:35.291623 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.291622    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-bundle\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:23:35.291804 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:35.291633    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd9ce54-76d8-499b-bb03-b0efe527fae4-util\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:23:36.013605 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:36.013568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng" event={"ID":"7fd9ce54-76d8-499b-bb03-b0efe527fae4","Type":"ContainerDied","Data":"7c61eb9ed79064d552b89f05722e9f8537c218792387e30ae7ecd0cb2278249a"}
Sep 05 02:23:36.013783 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:36.013617    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c61eb9ed79064d552b89f05722e9f8537c218792387e30ae7ecd0cb2278249a"
Sep 05 02:23:36.013783 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:36.013580    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abvsng"
Sep 05 02:23:39.746935 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.746898    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"]
Sep 05 02:23:39.747358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.747195    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerName="pull"
Sep 05 02:23:39.747358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.747206    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerName="pull"
Sep 05 02:23:39.747358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.747215    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerName="util"
Sep 05 02:23:39.747358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.747221    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerName="util"
Sep 05 02:23:39.747358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.747230    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerName="extract"
Sep 05 02:23:39.747358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.747235    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerName="extract"
Sep 05 02:23:39.747358 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.747287    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd9ce54-76d8-499b-bb03-b0efe527fae4" containerName="extract"
Sep 05 02:23:39.767210 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.767183    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"]
Sep 05 02:23:39.767374 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.767334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:39.771824 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.771801    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 05 02:23:39.772166 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.772150    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 05 02:23:39.772939 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.772357    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 05 02:23:39.773035 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.772451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-97h5m\""
Sep 05 02:23:39.830463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.830432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/17bb242e-f84d-4f34-8075-4d6748ccbb58-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-npmqm\" (UID: \"17bb242e-f84d-4f34-8075-4d6748ccbb58\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:39.830714 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.830543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhrg\" (UniqueName: \"kubernetes.io/projected/17bb242e-f84d-4f34-8075-4d6748ccbb58-kube-api-access-6vhrg\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-npmqm\" (UID: \"17bb242e-f84d-4f34-8075-4d6748ccbb58\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:39.931960 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.931924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhrg\" (UniqueName: \"kubernetes.io/projected/17bb242e-f84d-4f34-8075-4d6748ccbb58-kube-api-access-6vhrg\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-npmqm\" (UID: \"17bb242e-f84d-4f34-8075-4d6748ccbb58\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:39.931960 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.931963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/17bb242e-f84d-4f34-8075-4d6748ccbb58-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-npmqm\" (UID: \"17bb242e-f84d-4f34-8075-4d6748ccbb58\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:39.934303 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.934282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/17bb242e-f84d-4f34-8075-4d6748ccbb58-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-npmqm\" (UID: \"17bb242e-f84d-4f34-8075-4d6748ccbb58\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:39.945790 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:39.945757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhrg\" (UniqueName: \"kubernetes.io/projected/17bb242e-f84d-4f34-8075-4d6748ccbb58-kube-api-access-6vhrg\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-npmqm\" (UID: \"17bb242e-f84d-4f34-8075-4d6748ccbb58\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:40.080997 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:40.080959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:40.384053 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:40.383981    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"]
Sep 05 02:23:40.387307 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:23:40.387275    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bb242e_f84d_4f34_8075_4d6748ccbb58.slice/crio-c4e143077dda89c954657dea7193799ecd72f45745b43247f8d110d15d613ed1 WatchSource:0}: Error finding container c4e143077dda89c954657dea7193799ecd72f45745b43247f8d110d15d613ed1: Status 404 returned error can't find the container with id c4e143077dda89c954657dea7193799ecd72f45745b43247f8d110d15d613ed1
Sep 05 02:23:41.033109 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:41.033071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm" event={"ID":"17bb242e-f84d-4f34-8075-4d6748ccbb58","Type":"ContainerStarted","Data":"c4e143077dda89c954657dea7193799ecd72f45745b43247f8d110d15d613ed1"}
Sep 05 02:23:44.685677 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.685626    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-frw59"]
Sep 05 02:23:44.703648 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.703609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-frw59"]
Sep 05 02:23:44.703836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.703755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.705984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.705962    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 05 02:23:44.706181 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.706168    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 05 02:23:44.707057 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.707034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9hxb2\""
Sep 05 02:23:44.772056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.772016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqg2\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-kube-api-access-wxqg2\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.772236 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.772075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.772236 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.772111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d7c81c22-45b8-445c-bd8d-704e25d982cf-cabundle0\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.873199 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.873169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqg2\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-kube-api-access-wxqg2\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.873199 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.873203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.873401 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.873241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d7c81c22-45b8-445c-bd8d-704e25d982cf-cabundle0\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.873401 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:44.873364    2579 secret.go:281] references non-existent secret key: ca.crt
Sep 05 02:23:44.873401 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:44.873381    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 05 02:23:44.873401 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:44.873390    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-frw59: references non-existent secret key: ca.crt
Sep 05 02:23:44.873556 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:44.873437    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates podName:d7c81c22-45b8-445c-bd8d-704e25d982cf nodeName:}" failed. No retries permitted until 2026-09-05 02:23:45.373422972 +0000 UTC m=+369.345726830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates") pod "keda-operator-579c8958f7-frw59" (UID: "d7c81c22-45b8-445c-bd8d-704e25d982cf") : references non-existent secret key: ca.crt
Sep 05 02:23:44.873933 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.873915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d7c81c22-45b8-445c-bd8d-704e25d982cf-cabundle0\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.883032 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.883011    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqg2\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-kube-api-access-wxqg2\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:44.956426 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.956339    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"]
Sep 05 02:23:44.977481 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.977451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"]
Sep 05 02:23:44.977635 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.977555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:44.979831 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:44.979809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 05 02:23:45.050996 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.050957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm" event={"ID":"17bb242e-f84d-4f34-8075-4d6748ccbb58","Type":"ContainerStarted","Data":"447585b3b0244e2d1053a49f5282ba851376fbd721c64175d9d3b0ec1f5e65d4"}
Sep 05 02:23:45.051173 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.051099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:23:45.070626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.070575    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm" podStartSLOduration=2.5108095820000003 podStartE2EDuration="6.070559544s" totalImagesPullingTime="3.559749962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:23:45.069795066 +0000 UTC m=+369.042098969" watchObservedRunningTime="2026-09-05 02:23:45.070559544 +0000 UTC m=+369.042863420"
Sep 05 02:23:45.075214 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.075191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fblb\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-kube-api-access-5fblb\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.075327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.075233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/07708cbe-45e4-4d1c-9c05-adaca55208c8-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.075327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.075258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.176546 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.176508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fblb\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-kube-api-access-5fblb\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.176760 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.176581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/07708cbe-45e4-4d1c-9c05-adaca55208c8-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.176760 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.176726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.176890 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.176864    2579 secret.go:281] references non-existent secret key: tls.crt
Sep 05 02:23:45.176929 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.176889    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 05 02:23:45.176929 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.176911    2579 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 05 02:23:45.176989 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.176933    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 05 02:23:45.177025 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.177012    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates podName:07708cbe-45e4-4d1c-9c05-adaca55208c8 nodeName:}" failed. No retries permitted until 2026-09-05 02:23:45.676992244 +0000 UTC m=+369.649296112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates") pod "keda-metrics-apiserver-77857b7b98-jslv2" (UID: "07708cbe-45e4-4d1c-9c05-adaca55208c8") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 05 02:23:45.177025 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.177015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/07708cbe-45e4-4d1c-9c05-adaca55208c8-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.187192 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.187159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fblb\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-kube-api-access-5fblb\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.378620 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.378577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:45.378884 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.378743    2579 secret.go:281] references non-existent secret key: ca.crt
Sep 05 02:23:45.378884 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.378764    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 05 02:23:45.378884 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.378773    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-frw59: references non-existent secret key: ca.crt
Sep 05 02:23:45.378884 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.378838    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates podName:d7c81c22-45b8-445c-bd8d-704e25d982cf nodeName:}" failed. No retries permitted until 2026-09-05 02:23:46.37882122 +0000 UTC m=+370.351125091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates") pod "keda-operator-579c8958f7-frw59" (UID: "d7c81c22-45b8-445c-bd8d-704e25d982cf") : references non-existent secret key: ca.crt
Sep 05 02:23:45.664482 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.664401    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-7zmbb"]
Sep 05 02:23:45.681812 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.681777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:45.681976 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.681945    2579 secret.go:281] references non-existent secret key: tls.crt
Sep 05 02:23:45.681976 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.681968    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 05 02:23:45.682087 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.681989    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2: references non-existent secret key: tls.crt
Sep 05 02:23:45.682087 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.682054    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates podName:07708cbe-45e4-4d1c-9c05-adaca55208c8 nodeName:}" failed. No retries permitted until 2026-09-05 02:23:46.682034502 +0000 UTC m=+370.654338370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates") pod "keda-metrics-apiserver-77857b7b98-jslv2" (UID: "07708cbe-45e4-4d1c-9c05-adaca55208c8") : references non-existent secret key: tls.crt
Sep 05 02:23:45.686778 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.686753    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-7zmbb"]
Sep 05 02:23:45.687171 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.686869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:45.690180 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.690149    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 05 02:23:45.783090 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.783060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-certificates\") pod \"keda-admission-6c4d864b86-7zmbb\" (UID: \"7ac17d57-ee13-44f4-8972-a49fbef51221\") " pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:45.783090 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.783093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtbl\" (UniqueName: \"kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-kube-api-access-hgtbl\") pod \"keda-admission-6c4d864b86-7zmbb\" (UID: \"7ac17d57-ee13-44f4-8972-a49fbef51221\") " pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:45.883761 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.883723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-certificates\") pod \"keda-admission-6c4d864b86-7zmbb\" (UID: \"7ac17d57-ee13-44f4-8972-a49fbef51221\") " pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:45.883761 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.883760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtbl\" (UniqueName: \"kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-kube-api-access-hgtbl\") pod \"keda-admission-6c4d864b86-7zmbb\" (UID: \"7ac17d57-ee13-44f4-8972-a49fbef51221\") " pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:45.883962 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.883875    2579 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 05 02:23:45.883962 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.883903    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-7zmbb: secret "keda-admission-webhooks-certs" not found
Sep 05 02:23:45.883962 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:23:45.883955    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-certificates podName:7ac17d57-ee13-44f4-8972-a49fbef51221 nodeName:}" failed. No retries permitted until 2026-09-05 02:23:46.383939107 +0000 UTC m=+370.356242978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-certificates") pod "keda-admission-6c4d864b86-7zmbb" (UID: "7ac17d57-ee13-44f4-8972-a49fbef51221") : secret "keda-admission-webhooks-certs" not found
Sep 05 02:23:45.892553 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:45.892524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtbl\" (UniqueName: \"kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-kube-api-access-hgtbl\") pod \"keda-admission-6c4d864b86-7zmbb\" (UID: \"7ac17d57-ee13-44f4-8972-a49fbef51221\") " pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:46.388342 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.388303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-certificates\") pod \"keda-admission-6c4d864b86-7zmbb\" (UID: \"7ac17d57-ee13-44f4-8972-a49fbef51221\") " pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:46.388553 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.388374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:46.390854 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.390829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7ac17d57-ee13-44f4-8972-a49fbef51221-certificates\") pod \"keda-admission-6c4d864b86-7zmbb\" (UID: \"7ac17d57-ee13-44f4-8972-a49fbef51221\") " pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:46.391005 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.390982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d7c81c22-45b8-445c-bd8d-704e25d982cf-certificates\") pod \"keda-operator-579c8958f7-frw59\" (UID: \"d7c81c22-45b8-445c-bd8d-704e25d982cf\") " pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:46.514885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.514844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:46.597121 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.597092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:46.691456 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.691418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:46.693860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.693835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/07708cbe-45e4-4d1c-9c05-adaca55208c8-certificates\") pod \"keda-metrics-apiserver-77857b7b98-jslv2\" (UID: \"07708cbe-45e4-4d1c-9c05-adaca55208c8\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:46.788293 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.788257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:46.805079 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.805047    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-frw59"]
Sep 05 02:23:46.808960 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:23:46.808931    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c81c22_45b8_445c_bd8d_704e25d982cf.slice/crio-a69787f20809b65d43ebea99e1725c46aedeafbaa7ef32c679c752eeacbc7da9 WatchSource:0}: Error finding container a69787f20809b65d43ebea99e1725c46aedeafbaa7ef32c679c752eeacbc7da9: Status 404 returned error can't find the container with id a69787f20809b65d43ebea99e1725c46aedeafbaa7ef32c679c752eeacbc7da9
Sep 05 02:23:46.870684 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:46.870644    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-7zmbb"]
Sep 05 02:23:46.873553 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:23:46.873523    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac17d57_ee13_44f4_8972_a49fbef51221.slice/crio-9f27f9a59479d2164f78f65e73984e010b3d032a23df1c3901d9c5381a4bcca6 WatchSource:0}: Error finding container 9f27f9a59479d2164f78f65e73984e010b3d032a23df1c3901d9c5381a4bcca6: Status 404 returned error can't find the container with id 9f27f9a59479d2164f78f65e73984e010b3d032a23df1c3901d9c5381a4bcca6
Sep 05 02:23:47.061369 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:47.061334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-7zmbb" event={"ID":"7ac17d57-ee13-44f4-8972-a49fbef51221","Type":"ContainerStarted","Data":"9f27f9a59479d2164f78f65e73984e010b3d032a23df1c3901d9c5381a4bcca6"}
Sep 05 02:23:47.062398 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:47.062377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-frw59" event={"ID":"d7c81c22-45b8-445c-bd8d-704e25d982cf","Type":"ContainerStarted","Data":"a69787f20809b65d43ebea99e1725c46aedeafbaa7ef32c679c752eeacbc7da9"}
Sep 05 02:23:47.068475 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:47.068454    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"]
Sep 05 02:23:47.069837 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:23:47.069811    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07708cbe_45e4_4d1c_9c05_adaca55208c8.slice/crio-d1b57775b9efcce7095e173677739402ec050c13c9e6381960ff355521f8ee4c WatchSource:0}: Error finding container d1b57775b9efcce7095e173677739402ec050c13c9e6381960ff355521f8ee4c: Status 404 returned error can't find the container with id d1b57775b9efcce7095e173677739402ec050c13c9e6381960ff355521f8ee4c
Sep 05 02:23:48.069550 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:48.069502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2" event={"ID":"07708cbe-45e4-4d1c-9c05-adaca55208c8","Type":"ContainerStarted","Data":"d1b57775b9efcce7095e173677739402ec050c13c9e6381960ff355521f8ee4c"}
Sep 05 02:23:49.076994 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:49.076953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-7zmbb" event={"ID":"7ac17d57-ee13-44f4-8972-a49fbef51221","Type":"ContainerStarted","Data":"5eb9faaf41d4c6902b32db7df0d577d4739958e691f71a575210e122d82b43f0"}
Sep 05 02:23:49.077459 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:49.077200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:23:49.095253 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:49.095191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-7zmbb" podStartSLOduration=2.756096249 podStartE2EDuration="4.095169837s" totalImagesPullingTime="1.339073588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:23:49.093591679 +0000 UTC m=+373.065895557" watchObservedRunningTime="2026-09-05 02:23:49.095169837 +0000 UTC m=+373.067473715"
Sep 05 02:23:51.087581 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:51.087546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2" event={"ID":"07708cbe-45e4-4d1c-9c05-adaca55208c8","Type":"ContainerStarted","Data":"6a60a33616f1207b9124c1d5f2968ae8484903f7cde797930e90eea3252a5a29"}
Sep 05 02:23:51.088031 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:51.087672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:23:51.089320 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:51.089293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-frw59" event={"ID":"d7c81c22-45b8-445c-bd8d-704e25d982cf","Type":"ContainerStarted","Data":"3c84a87b7ced1ebdad0d4b57ca217e21ddf3e3dc1ccf4a3fe3f29ba788734ae7"}
Sep 05 02:23:51.089456 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:51.089432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:23:51.106102 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:51.106054    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2" podStartSLOduration=3.24236982 podStartE2EDuration="7.106041147s" totalImagesPullingTime="3.863671327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:23:51.10513793 +0000 UTC m=+375.077441807" watchObservedRunningTime="2026-09-05 02:23:51.106041147 +0000 UTC m=+375.078345022"
Sep 05 02:23:51.123580 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:23:51.123536    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-frw59" podStartSLOduration=2.998890345 podStartE2EDuration="7.123523565s" totalImagesPullingTime="4.12463322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:23:51.122575963 +0000 UTC m=+375.094879869" watchObservedRunningTime="2026-09-05 02:23:51.123523565 +0000 UTC m=+375.095827441"
Sep 05 02:24:02.100011 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:02.099979    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-jslv2"
Sep 05 02:24:06.057356 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:06.057321    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-npmqm"
Sep 05 02:24:10.084370 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:10.084341    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-7zmbb"
Sep 05 02:24:12.097360 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:12.097322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-frw59"
Sep 05 02:24:32.245114 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.245082    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9b6d7b79-72kh2"]
Sep 05 02:24:32.247293 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.247278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.259438 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.259413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9b6d7b79-72kh2"]
Sep 05 02:24:32.377039 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.377001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-oauth-config\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.377039 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.377044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-serving-cert\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.377250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.377063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-config\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.377250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.377150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-service-ca\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.377250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.377184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5956x\" (UniqueName: \"kubernetes.io/projected/7916ab3d-d846-4260-a6f6-56ffcbd18771-kube-api-access-5956x\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.377250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.377206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-trusted-ca-bundle\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.377384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.377252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-oauth-serving-cert\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.477715 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.477652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-oauth-config\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.477715 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.477713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-serving-cert\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.477943 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.477732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-config\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.477943 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.477768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-service-ca\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.477943 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.477795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5956x\" (UniqueName: \"kubernetes.io/projected/7916ab3d-d846-4260-a6f6-56ffcbd18771-kube-api-access-5956x\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.477943 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.477822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-trusted-ca-bundle\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.477943 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.477868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-oauth-serving-cert\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.478746 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.478717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-config\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.478746 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.478733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-oauth-serving-cert\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.478939 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.478717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-service-ca\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.478939 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.478851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7916ab3d-d846-4260-a6f6-56ffcbd18771-trusted-ca-bundle\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.480236 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.480205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-oauth-config\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.480336 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.480309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7916ab3d-d846-4260-a6f6-56ffcbd18771-console-serving-cert\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.486022 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.486000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5956x\" (UniqueName: \"kubernetes.io/projected/7916ab3d-d846-4260-a6f6-56ffcbd18771-kube-api-access-5956x\") pod \"console-7c9b6d7b79-72kh2\" (UID: \"7916ab3d-d846-4260-a6f6-56ffcbd18771\") " pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.556872 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.556841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:32.840189 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:32.840108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9b6d7b79-72kh2"]
Sep 05 02:24:32.844055 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:24:32.844030    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7916ab3d_d846_4260_a6f6_56ffcbd18771.slice/crio-d36985e664829a107fb54511924261c02744b99d98acc0bfb50759bae6f5ba09 WatchSource:0}: Error finding container d36985e664829a107fb54511924261c02744b99d98acc0bfb50759bae6f5ba09: Status 404 returned error can't find the container with id d36985e664829a107fb54511924261c02744b99d98acc0bfb50759bae6f5ba09
Sep 05 02:24:33.293363 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:33.293329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9b6d7b79-72kh2" event={"ID":"7916ab3d-d846-4260-a6f6-56ffcbd18771","Type":"ContainerStarted","Data":"4e82e1d9c9c1a8ea84ed3f2f9c325e958fee3a5eb6537a20e5ed44bd51be731e"}
Sep 05 02:24:33.293825 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:33.293374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9b6d7b79-72kh2" event={"ID":"7916ab3d-d846-4260-a6f6-56ffcbd18771","Type":"ContainerStarted","Data":"d36985e664829a107fb54511924261c02744b99d98acc0bfb50759bae6f5ba09"}
Sep 05 02:24:33.318422 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:33.318375    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9b6d7b79-72kh2" podStartSLOduration=1.31836214 podStartE2EDuration="1.31836214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:24:33.318067152 +0000 UTC m=+417.290371028" watchObservedRunningTime="2026-09-05 02:24:33.31836214 +0000 UTC m=+417.290666017"
Sep 05 02:24:42.557161 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:42.557126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:42.557593 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:42.557172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:42.562070 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:42.562048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:43.338882 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:43.338852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9b6d7b79-72kh2"
Sep 05 02:24:43.406644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:24:43.406611    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-656c55dc96-qlhhx"]
Sep 05 02:25:06.698482 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.698405    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-r2hv6"]
Sep 05 02:25:06.701651 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.701630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:06.704179 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.704155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 05 02:25:06.704291 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.704200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9qdjn\""
Sep 05 02:25:06.704835 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.704822    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 05 02:25:06.704889 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.704822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 05 02:25:06.714698 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.714677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-r2hv6"]
Sep 05 02:25:06.858415 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.858380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/574c5e5a-a96e-42d7-b1f5-eb9df2e315fb-data\") pod \"seaweedfs-64568bcd49-r2hv6\" (UID: \"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb\") " pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:06.858592 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.858423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzpn\" (UniqueName: \"kubernetes.io/projected/574c5e5a-a96e-42d7-b1f5-eb9df2e315fb-kube-api-access-xdzpn\") pod \"seaweedfs-64568bcd49-r2hv6\" (UID: \"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb\") " pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:06.959835 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.959737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzpn\" (UniqueName: \"kubernetes.io/projected/574c5e5a-a96e-42d7-b1f5-eb9df2e315fb-kube-api-access-xdzpn\") pod \"seaweedfs-64568bcd49-r2hv6\" (UID: \"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb\") " pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:06.960007 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.959862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/574c5e5a-a96e-42d7-b1f5-eb9df2e315fb-data\") pod \"seaweedfs-64568bcd49-r2hv6\" (UID: \"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb\") " pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:06.960252 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.960230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/574c5e5a-a96e-42d7-b1f5-eb9df2e315fb-data\") pod \"seaweedfs-64568bcd49-r2hv6\" (UID: \"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb\") " pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:06.972335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:06.972314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzpn\" (UniqueName: \"kubernetes.io/projected/574c5e5a-a96e-42d7-b1f5-eb9df2e315fb-kube-api-access-xdzpn\") pod \"seaweedfs-64568bcd49-r2hv6\" (UID: \"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb\") " pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:07.011144 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:07.011106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:07.283940 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:07.283854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-r2hv6"]
Sep 05 02:25:07.286637 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:25:07.286594    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574c5e5a_a96e_42d7_b1f5_eb9df2e315fb.slice/crio-1f7d7b27deae8bd97ab66774060c496abf1462ff91762c900ec68fefe6445b86 WatchSource:0}: Error finding container 1f7d7b27deae8bd97ab66774060c496abf1462ff91762c900ec68fefe6445b86: Status 404 returned error can't find the container with id 1f7d7b27deae8bd97ab66774060c496abf1462ff91762c900ec68fefe6445b86
Sep 05 02:25:07.441097 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:07.441062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-r2hv6" event={"ID":"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb","Type":"ContainerStarted","Data":"1f7d7b27deae8bd97ab66774060c496abf1462ff91762c900ec68fefe6445b86"}
Sep 05 02:25:08.433129 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.433084    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-656c55dc96-qlhhx" podUID="cf7d1add-d204-4141-a325-c32d7678414c" containerName="console" containerID="cri-o://fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad" gracePeriod=15
Sep 05 02:25:08.504356 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.504318    2579 patch_prober.go:28] interesting pod/console-656c55dc96-qlhhx container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.25:8443/health\": dial tcp 10.132.0.25:8443: connect: connection refused" start-of-body=
Sep 05 02:25:08.504534 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.504380    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-656c55dc96-qlhhx" podUID="cf7d1add-d204-4141-a325-c32d7678414c" containerName="console" probeResult="failure" output="Get \"https://10.132.0.25:8443/health\": dial tcp 10.132.0.25:8443: connect: connection refused"
Sep 05 02:25:08.707166 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.707133    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:25:08.775124 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-trusted-ca-bundle\") pod \"cf7d1add-d204-4141-a325-c32d7678414c\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") "
Sep 05 02:25:08.775124 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-console-config\") pod \"cf7d1add-d204-4141-a325-c32d7678414c\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") "
Sep 05 02:25:08.775362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775171    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-serving-cert\") pod \"cf7d1add-d204-4141-a325-c32d7678414c\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") "
Sep 05 02:25:08.775362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775199    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-oauth-serving-cert\") pod \"cf7d1add-d204-4141-a325-c32d7678414c\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") "
Sep 05 02:25:08.775362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-service-ca\") pod \"cf7d1add-d204-4141-a325-c32d7678414c\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") "
Sep 05 02:25:08.775362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775254    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-oauth-config\") pod \"cf7d1add-d204-4141-a325-c32d7678414c\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") "
Sep 05 02:25:08.775362 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d68xf\" (UniqueName: \"kubernetes.io/projected/cf7d1add-d204-4141-a325-c32d7678414c-kube-api-access-d68xf\") pod \"cf7d1add-d204-4141-a325-c32d7678414c\" (UID: \"cf7d1add-d204-4141-a325-c32d7678414c\") "
Sep 05 02:25:08.775642 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-console-config" (OuterVolumeSpecName: "console-config") pod "cf7d1add-d204-4141-a325-c32d7678414c" (UID: "cf7d1add-d204-4141-a325-c32d7678414c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:08.775845 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-service-ca" (OuterVolumeSpecName: "service-ca") pod "cf7d1add-d204-4141-a325-c32d7678414c" (UID: "cf7d1add-d204-4141-a325-c32d7678414c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:08.775933 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775873    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cf7d1add-d204-4141-a325-c32d7678414c" (UID: "cf7d1add-d204-4141-a325-c32d7678414c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:08.775933 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.775890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cf7d1add-d204-4141-a325-c32d7678414c" (UID: "cf7d1add-d204-4141-a325-c32d7678414c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:25:08.777897 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.777872    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cf7d1add-d204-4141-a325-c32d7678414c" (UID: "cf7d1add-d204-4141-a325-c32d7678414c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:25:08.777993 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.777917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7d1add-d204-4141-a325-c32d7678414c-kube-api-access-d68xf" (OuterVolumeSpecName: "kube-api-access-d68xf") pod "cf7d1add-d204-4141-a325-c32d7678414c" (UID: "cf7d1add-d204-4141-a325-c32d7678414c"). InnerVolumeSpecName "kube-api-access-d68xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:25:08.778058 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.778004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cf7d1add-d204-4141-a325-c32d7678414c" (UID: "cf7d1add-d204-4141-a325-c32d7678414c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:25:08.876511 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.876412    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:25:08.876511 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.876497    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-oauth-serving-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:25:08.876511 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.876517    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-service-ca\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:25:08.877307 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.876531    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cf7d1add-d204-4141-a325-c32d7678414c-console-oauth-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:25:08.877307 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.876581    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68xf\" (UniqueName: \"kubernetes.io/projected/cf7d1add-d204-4141-a325-c32d7678414c-kube-api-access-d68xf\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:25:08.877307 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.876600    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-trusted-ca-bundle\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:25:08.877307 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:08.876619    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cf7d1add-d204-4141-a325-c32d7678414c-console-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:25:09.458162 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.458056    2579 generic.go:356] "Generic (PLEG): container finished" podID="cf7d1add-d204-4141-a325-c32d7678414c" containerID="fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad" exitCode=0
Sep 05 02:25:09.458162 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.458138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656c55dc96-qlhhx" event={"ID":"cf7d1add-d204-4141-a325-c32d7678414c","Type":"ContainerDied","Data":"fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad"}
Sep 05 02:25:09.459268 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.458279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656c55dc96-qlhhx" event={"ID":"cf7d1add-d204-4141-a325-c32d7678414c","Type":"ContainerDied","Data":"084dd2533599544973f6f224d4aaef51a07dd86cd41ded45ff95a40e5ef5bc5e"}
Sep 05 02:25:09.459268 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.458318    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656c55dc96-qlhhx"
Sep 05 02:25:09.459268 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.458327    2579 scope.go:117] "RemoveContainer" containerID="fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad"
Sep 05 02:25:09.479143 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.479102    2579 scope.go:117] "RemoveContainer" containerID="fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad"
Sep 05 02:25:09.480092 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:25:09.480027    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad\": container with ID starting with fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad not found: ID does not exist" containerID="fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad"
Sep 05 02:25:09.480333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.480118    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad"} err="failed to get container status \"fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad\": rpc error: code = NotFound desc = could not find container \"fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad\": container with ID starting with fb5b82e1376c9c4bfc66ebe8841f434630f2db3aad73aae508b5b62735efc3ad not found: ID does not exist"
Sep 05 02:25:09.491411 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.491332    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-656c55dc96-qlhhx"]
Sep 05 02:25:09.504119 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:09.496884    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-656c55dc96-qlhhx"]
Sep 05 02:25:10.556838 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:10.556730    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7d1add-d204-4141-a325-c32d7678414c" path="/var/lib/kubelet/pods/cf7d1add-d204-4141-a325-c32d7678414c/volumes"
Sep 05 02:25:11.480038 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:11.479945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-r2hv6" event={"ID":"574c5e5a-a96e-42d7-b1f5-eb9df2e315fb","Type":"ContainerStarted","Data":"706386c0d475a83248147f9aa93af48d4a4a1dc44519c7cbc0083b4935016046"}
Sep 05 02:25:11.480531 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:11.480128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:25:11.501844 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:11.501567    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-r2hv6" podStartSLOduration=1.54442262 podStartE2EDuration="5.50152652s" totalImagesPullingTime="3.9571039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:25:11.498860487 +0000 UTC m=+455.471164364" watchObservedRunningTime="2026-09-05 02:25:11.50152652 +0000 UTC m=+455.473830480"
Sep 05 02:25:17.487194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:25:17.487161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-r2hv6"
Sep 05 02:26:29.339571 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.339366    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-djbsn"]
Sep 05 02:26:29.340551 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.340444    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7d1add-d204-4141-a325-c32d7678414c" containerName="console"
Sep 05 02:26:29.340551 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.340498    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7d1add-d204-4141-a325-c32d7678414c" containerName="console"
Sep 05 02:26:29.340741 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.340654    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7d1add-d204-4141-a325-c32d7678414c" containerName="console"
Sep 05 02:26:29.345375 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.345312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.347985 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.347930    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-jj6d9"]
Sep 05 02:26:29.348272 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.348102    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 05 02:26:29.348272 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.348146    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-j6ltf\""
Sep 05 02:26:29.352860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.352815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.357741 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.357701    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-ldx9t\""
Sep 05 02:26:29.358091 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.357701    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 05 02:26:29.358434 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.357851    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 05 02:26:29.360637 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.360444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-djbsn"]
Sep 05 02:26:29.364385 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.364339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-jj6d9"]
Sep 05 02:26:29.499366 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.499280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-cert\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.499886 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.499423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-metrics-certs\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.499886 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.499585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzqv\" (UniqueName: \"kubernetes.io/projected/4dec06be-89c7-4722-bdee-0b0a2764de45-kube-api-access-6nzqv\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.499886 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.499760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac2fa516-9ee4-4e90-a6cc-eff78d8e250d-tls-certs\") pod \"model-serving-api-7b954ffc95-djbsn\" (UID: \"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d\") " pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.499886 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.499815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrffd\" (UniqueName: \"kubernetes.io/projected/ac2fa516-9ee4-4e90-a6cc-eff78d8e250d-kube-api-access-qrffd\") pod \"model-serving-api-7b954ffc95-djbsn\" (UID: \"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d\") " pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.600898 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.600639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-cert\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.600898 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.600783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-metrics-certs\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.600898 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.600868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzqv\" (UniqueName: \"kubernetes.io/projected/4dec06be-89c7-4722-bdee-0b0a2764de45-kube-api-access-6nzqv\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.600898 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:26:29.600901    2579 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 05 02:26:29.601515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.600950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac2fa516-9ee4-4e90-a6cc-eff78d8e250d-tls-certs\") pod \"model-serving-api-7b954ffc95-djbsn\" (UID: \"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d\") " pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.601515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.600981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrffd\" (UniqueName: \"kubernetes.io/projected/ac2fa516-9ee4-4e90-a6cc-eff78d8e250d-kube-api-access-qrffd\") pod \"model-serving-api-7b954ffc95-djbsn\" (UID: \"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d\") " pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.601515 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:26:29.601070    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-cert podName:4dec06be-89c7-4722-bdee-0b0a2764de45 nodeName:}" failed. No retries permitted until 2026-09-05 02:26:30.101022006 +0000 UTC m=+534.073325876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-cert") pod "odh-model-controller-7b966d7984-jj6d9" (UID: "4dec06be-89c7-4722-bdee-0b0a2764de45") : secret "odh-model-controller-webhook-cert" not found
Sep 05 02:26:29.604735 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.604694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac2fa516-9ee4-4e90-a6cc-eff78d8e250d-tls-certs\") pod \"model-serving-api-7b954ffc95-djbsn\" (UID: \"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d\") " pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.604925 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.604755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-metrics-certs\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.611976 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.611921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzqv\" (UniqueName: \"kubernetes.io/projected/4dec06be-89c7-4722-bdee-0b0a2764de45-kube-api-access-6nzqv\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:29.611976 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.611977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrffd\" (UniqueName: \"kubernetes.io/projected/ac2fa516-9ee4-4e90-a6cc-eff78d8e250d-kube-api-access-qrffd\") pod \"model-serving-api-7b954ffc95-djbsn\" (UID: \"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d\") " pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.668565 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.668476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:29.988737 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:29.988343    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-djbsn"]
Sep 05 02:26:29.994289 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:26:29.994233    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2fa516_9ee4_4e90_a6cc_eff78d8e250d.slice/crio-99cae585d0f038780078c4a8a4feb1598aee0dcadfa96fec914bff04c67f5085 WatchSource:0}: Error finding container 99cae585d0f038780078c4a8a4feb1598aee0dcadfa96fec914bff04c67f5085: Status 404 returned error can't find the container with id 99cae585d0f038780078c4a8a4feb1598aee0dcadfa96fec914bff04c67f5085
Sep 05 02:26:30.107253 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:30.107177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-cert\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:30.111305 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:30.111269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dec06be-89c7-4722-bdee-0b0a2764de45-cert\") pod \"odh-model-controller-7b966d7984-jj6d9\" (UID: \"4dec06be-89c7-4722-bdee-0b0a2764de45\") " pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:30.281566 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:30.281512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:30.562186 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:30.562164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-jj6d9"]
Sep 05 02:26:30.564608 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:26:30.564574    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dec06be_89c7_4722_bdee_0b0a2764de45.slice/crio-bf415383d3db3b463de772567da3bfd4a936c66ae570aa0c2a42260b597b7d3b WatchSource:0}: Error finding container bf415383d3db3b463de772567da3bfd4a936c66ae570aa0c2a42260b597b7d3b: Status 404 returned error can't find the container with id bf415383d3db3b463de772567da3bfd4a936c66ae570aa0c2a42260b597b7d3b
Sep 05 02:26:30.836243 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:30.836160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-jj6d9" event={"ID":"4dec06be-89c7-4722-bdee-0b0a2764de45","Type":"ContainerStarted","Data":"bf415383d3db3b463de772567da3bfd4a936c66ae570aa0c2a42260b597b7d3b"}
Sep 05 02:26:30.837514 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:30.837485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-djbsn" event={"ID":"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d","Type":"ContainerStarted","Data":"99cae585d0f038780078c4a8a4feb1598aee0dcadfa96fec914bff04c67f5085"}
Sep 05 02:26:31.844098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:31.844066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-djbsn" event={"ID":"ac2fa516-9ee4-4e90-a6cc-eff78d8e250d","Type":"ContainerStarted","Data":"cd51e2cc36d2539a6d3cc1694aa3a2530c1684905b02ba10f0e7680f747b8dd1"}
Sep 05 02:26:31.844505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:31.844181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:31.868066 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:31.867928    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-djbsn" podStartSLOduration=1.236440942 podStartE2EDuration="2.867908567s" totalImagesPullingTime="1.631467625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:31.867417685 +0000 UTC m=+535.839721561" watchObservedRunningTime="2026-09-05 02:26:31.867908567 +0000 UTC m=+535.840212444"
Sep 05 02:26:33.856078 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:33.856042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-jj6d9" event={"ID":"4dec06be-89c7-4722-bdee-0b0a2764de45","Type":"ContainerStarted","Data":"40a601c2b5988e40fad64a496ff2e940e7a5be69a356e36901658a86c481e11c"}
Sep 05 02:26:33.856450 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:33.856115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:26:33.874500 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:33.874456    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-jj6d9" podStartSLOduration=2.368753909 podStartE2EDuration="4.874440415s" totalImagesPullingTime="2.505686506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:26:33.873136212 +0000 UTC m=+537.845440090" watchObservedRunningTime="2026-09-05 02:26:33.874440415 +0000 UTC m=+537.846744291"
Sep 05 02:26:42.854374 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:42.854343    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-djbsn"
Sep 05 02:26:44.862754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:26:44.862720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-jj6d9"
Sep 05 02:28:16.899508 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.899471    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"]
Sep 05 02:28:16.902956 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.902938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:16.908242 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.908220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:28:16.908896 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.908871    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0d68f-kube-rbac-proxy-sar-config\""
Sep 05 02:28:16.909123 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.909108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0d68f-predictor-serving-cert\""
Sep 05 02:28:16.909937 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.909919    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:28:16.910051 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.910033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:28:16.910491 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.910476    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:28:16.918808 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.918790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:28:16.957463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:16.957430    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"]
Sep 05 02:28:17.008785 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.008747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.009017 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.008797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b6f4cc8-9215-4535-a431-26dd66f663d9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.009017 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.008858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b6f4cc8-9215-4535-a431-26dd66f663d9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.009017 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.008909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.009017 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.008960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxssj\" (UniqueName: \"kubernetes.io/projected/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-api-access-jxssj\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.110194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.110158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b6f4cc8-9215-4535-a431-26dd66f663d9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.110194 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.110199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b6f4cc8-9215-4535-a431-26dd66f663d9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.110468 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.110224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.110468 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.110256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxssj\" (UniqueName: \"kubernetes.io/projected/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-api-access-jxssj\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.110468 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.110308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.110711 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.110685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b6f4cc8-9215-4535-a431-26dd66f663d9-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.111034 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.111014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.111152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.111132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.112671 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.112639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b6f4cc8-9215-4535-a431-26dd66f663d9-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.145732 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.145691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxssj\" (UniqueName: \"kubernetes.io/projected/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-api-access-jxssj\") pod \"isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.215460 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.215385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:17.556022 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:28:17.555991    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6f4cc8_9215_4535_a431_26dd66f663d9.slice/crio-ecc02530de935fa55817618bb94c871f004790454f073be3de28b0e91d50b37e WatchSource:0}: Error finding container ecc02530de935fa55817618bb94c871f004790454f073be3de28b0e91d50b37e: Status 404 returned error can't find the container with id ecc02530de935fa55817618bb94c871f004790454f073be3de28b0e91d50b37e
Sep 05 02:28:17.556516 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:17.556490    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"]
Sep 05 02:28:18.337263 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:18.337232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerStarted","Data":"ecc02530de935fa55817618bb94c871f004790454f073be3de28b0e91d50b37e"}
Sep 05 02:28:21.357111 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:21.357072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerStarted","Data":"972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f"}
Sep 05 02:28:25.375834 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:25.375802    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerID="972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f" exitCode=0
Sep 05 02:28:25.376232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:25.375853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerDied","Data":"972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f"}
Sep 05 02:28:25.376977 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:25.376958    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:28:39.468067 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:39.467983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerStarted","Data":"557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6"}
Sep 05 02:28:41.482288 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:41.482254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerStarted","Data":"6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63"}
Sep 05 02:28:43.501020 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:43.500919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerStarted","Data":"31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a"}
Sep 05 02:28:43.501931 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:43.501403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:43.527461 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:43.527316    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podStartSLOduration=2.447490411 podStartE2EDuration="27.527269262s" totalImagesPullingTime="21.133812521s" totalInitContainerRuntime="3.94596633s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:28:16 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:28:43.524576933 +0000 UTC m=+667.496880809" watchObservedRunningTime="2026-09-05 02:28:43.527269262 +0000 UTC m=+667.499573139"
Sep 05 02:28:44.513387 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:44.513296    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:44.514817 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:44.514771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:44.520513 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:44.520417    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:28:44.521749 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:44.521596    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:28:45.518576 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:45.518478    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:28:45.519642 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:45.519419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:28:45.524314 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:45.524253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:28:46.522029 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:46.521985    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:28:46.522495 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:46.522378    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:28:56.522548 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:56.522501    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:28:56.523004 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:28:56.522840    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:29:06.522644 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:06.522595    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:29:06.523110 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:06.523086    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:29:16.522455 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:16.522411    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:29:16.522956 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:16.522853    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:29:26.522632 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:26.522582    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:29:26.523145 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:26.523057    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:29:36.522631 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:36.522544    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:29:36.523064 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:36.522966    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:29:46.522143 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:46.522093    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:29:46.522537 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:46.522458    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:29:56.522603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:56.522569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:29:56.523059 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:29:56.522813    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:30:11.743890 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:11.743851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"]
Sep 05 02:30:11.744294 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:11.744148    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" containerID="cri-o://557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6" gracePeriod=30
Sep 05 02:30:11.744294 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:11.744174    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy" containerID="cri-o://6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63" gracePeriod=30
Sep 05 02:30:11.744294 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:11.744168    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" containerID="cri-o://31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a" gracePeriod=30
Sep 05 02:30:11.929800 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:11.929762    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerID="6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63" exitCode=2
Sep 05 02:30:11.929976 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:11.929842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerDied","Data":"6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63"}
Sep 05 02:30:12.029445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.029410    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"]
Sep 05 02:30:12.033239 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.033219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.037478 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.037460    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e3535-kube-rbac-proxy-sar-config\""
Sep 05 02:30:12.039459 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.039264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e3535-predictor-serving-cert\""
Sep 05 02:30:12.051924 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.051903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"]
Sep 05 02:30:12.129036 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.129003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wh6k\" (UniqueName: \"kubernetes.io/projected/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-api-access-5wh6k\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.129186 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.129046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.129186 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.129103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.129186 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.129156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.129186 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.129177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b08ef4-da4f-47df-8c1b-99981f2b1749-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.230147 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230106    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wh6k\" (UniqueName: \"kubernetes.io/projected/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-api-access-5wh6k\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.230333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.230333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.230333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.230333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b08ef4-da4f-47df-8c1b-99981f2b1749-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.230591 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.230954 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.231035 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.230979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.232839 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.232818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b08ef4-da4f-47df-8c1b-99981f2b1749-proxy-tls\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.237652 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.237635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wh6k\" (UniqueName: \"kubernetes.io/projected/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-api-access-5wh6k\") pod \"isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.343718 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.343591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:12.618688 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.618646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"]
Sep 05 02:30:12.621065 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:30:12.621035    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b08ef4_da4f_47df_8c1b_99981f2b1749.slice/crio-498225ba443ef8a86dba0391071498572411cbffb3630a30e4cb6f58c37f565c WatchSource:0}: Error finding container 498225ba443ef8a86dba0391071498572411cbffb3630a30e4cb6f58c37f565c: Status 404 returned error can't find the container with id 498225ba443ef8a86dba0391071498572411cbffb3630a30e4cb6f58c37f565c
Sep 05 02:30:12.936409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.936312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerStarted","Data":"ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7"}
Sep 05 02:30:12.936409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:12.936385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerStarted","Data":"498225ba443ef8a86dba0391071498572411cbffb3630a30e4cb6f58c37f565c"}
Sep 05 02:30:15.519203 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:15.519103    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:30:16.522792 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:16.522738    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:16.523256 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:16.523093    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:16.969239 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:16.969153    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerID="ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7" exitCode=0
Sep 05 02:30:16.969239 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:16.969227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerDied","Data":"ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7"}
Sep 05 02:30:20.518311 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:20.518259    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:30:22.001410 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:22.001376    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerID="557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6" exitCode=0
Sep 05 02:30:22.001962 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:22.001466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerDied","Data":"557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6"}
Sep 05 02:30:25.518414 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:25.518370    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:30:25.518834 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:25.518512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:30:26.522272 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:26.522230    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:26.522743 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:26.522589    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:30.518581 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:30.518530    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:30:35.518295 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:35.518253    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:30:36.078997 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.078960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerStarted","Data":"7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228"}
Sep 05 02:30:36.079163 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.079047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerStarted","Data":"650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6"}
Sep 05 02:30:36.079357 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.079332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:36.100885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.100830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podStartSLOduration=2.087075007 podStartE2EDuration="24.100813919s" totalImagesPullingTime="18.004183974s" totalInitContainerRuntime="4.009554938s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:30:36.098866423 +0000 UTC m=+780.071170301" watchObservedRunningTime="2026-09-05 02:30:36.100813919 +0000 UTC m=+780.073117794"
Sep 05 02:30:36.522448 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.522409    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 05 02:30:36.522844 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.522547    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:30:36.522844 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.522799    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 02:30:36.522972 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:36.522889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:30:37.083343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:37.083315    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:37.084613 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:37.084580    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:30:38.087904 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:38.087865    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:30:40.518437 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:40.518392    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 05 02:30:41.889376 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.889353    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:30:41.994430 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b6f4cc8-9215-4535-a431-26dd66f663d9-proxy-tls\") pod \"3b6f4cc8-9215-4535-a431-26dd66f663d9\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") "
Sep 05 02:30:41.994607 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994444    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-rbac-proxy-sar-config\") pod \"3b6f4cc8-9215-4535-a431-26dd66f663d9\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") "
Sep 05 02:30:41.994607 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxssj\" (UniqueName: \"kubernetes.io/projected/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-api-access-jxssj\") pod \"3b6f4cc8-9215-4535-a431-26dd66f663d9\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") "
Sep 05 02:30:41.994607 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-cabundle-cert\") pod \"3b6f4cc8-9215-4535-a431-26dd66f663d9\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") "
Sep 05 02:30:41.994607 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b6f4cc8-9215-4535-a431-26dd66f663d9-kserve-provision-location\") pod \"3b6f4cc8-9215-4535-a431-26dd66f663d9\" (UID: \"3b6f4cc8-9215-4535-a431-26dd66f663d9\") "
Sep 05 02:30:41.994953 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994923    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6f4cc8-9215-4535-a431-26dd66f663d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b6f4cc8-9215-4535-a431-26dd66f663d9" (UID: "3b6f4cc8-9215-4535-a431-26dd66f663d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:30:41.995044 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b6f4cc8-9215-4535-a431-26dd66f663d9" (UID: "3b6f4cc8-9215-4535-a431-26dd66f663d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:41.995044 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.994921    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b6f4cc8-9215-4535-a431-26dd66f663d9" (UID: "3b6f4cc8-9215-4535-a431-26dd66f663d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:30:41.996695 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.996671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6f4cc8-9215-4535-a431-26dd66f663d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b6f4cc8-9215-4535-a431-26dd66f663d9" (UID: "3b6f4cc8-9215-4535-a431-26dd66f663d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:30:41.996792 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:41.996757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-api-access-jxssj" (OuterVolumeSpecName: "kube-api-access-jxssj") pod "3b6f4cc8-9215-4535-a431-26dd66f663d9" (UID: "3b6f4cc8-9215-4535-a431-26dd66f663d9"). InnerVolumeSpecName "kube-api-access-jxssj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:30:42.095537 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.095505    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b6f4cc8-9215-4535-a431-26dd66f663d9-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:30:42.095537 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.095536    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:30:42.095746 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.095547    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxssj\" (UniqueName: \"kubernetes.io/projected/3b6f4cc8-9215-4535-a431-26dd66f663d9-kube-api-access-jxssj\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:30:42.095746 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.095556    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b6f4cc8-9215-4535-a431-26dd66f663d9-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:30:42.095746 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.095565    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b6f4cc8-9215-4535-a431-26dd66f663d9-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:30:42.108207 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.108179    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerID="31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a" exitCode=0
Sep 05 02:30:42.108316 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.108258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerDied","Data":"31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a"}
Sep 05 02:30:42.108371 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.108271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"
Sep 05 02:30:42.108409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.108372    2579 scope.go:117] "RemoveContainer" containerID="31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a"
Sep 05 02:30:42.108441 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.108358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx" event={"ID":"3b6f4cc8-9215-4535-a431-26dd66f663d9","Type":"ContainerDied","Data":"ecc02530de935fa55817618bb94c871f004790454f073be3de28b0e91d50b37e"}
Sep 05 02:30:42.117408 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.117387    2579 scope.go:117] "RemoveContainer" containerID="6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63"
Sep 05 02:30:42.124712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.124693    2579 scope.go:117] "RemoveContainer" containerID="557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6"
Sep 05 02:30:42.131022 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.131001    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"]
Sep 05 02:30:42.132330 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.132306    2579 scope.go:117] "RemoveContainer" containerID="972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f"
Sep 05 02:30:42.134513 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.134494    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0d68f-predictor-645bf449f-25bxx"]
Sep 05 02:30:42.139965 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.139950    2579 scope.go:117] "RemoveContainer" containerID="31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a"
Sep 05 02:30:42.140211 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:30:42.140193    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a\": container with ID starting with 31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a not found: ID does not exist" containerID="31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a"
Sep 05 02:30:42.140256 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.140220    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a"} err="failed to get container status \"31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a\": rpc error: code = NotFound desc = could not find container \"31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a\": container with ID starting with 31e5d04c63781190c6a35613b8ce113a5c6b2fd23608c678a6c399dc77221d5a not found: ID does not exist"
Sep 05 02:30:42.140256 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.140236    2579 scope.go:117] "RemoveContainer" containerID="6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63"
Sep 05 02:30:42.140476 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:30:42.140442    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63\": container with ID starting with 6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63 not found: ID does not exist" containerID="6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63"
Sep 05 02:30:42.140515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.140482    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63"} err="failed to get container status \"6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63\": rpc error: code = NotFound desc = could not find container \"6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63\": container with ID starting with 6d090c2203a23604f88c241db8438339eb3907d1c85cbf28fe9be42eb9698d63 not found: ID does not exist"
Sep 05 02:30:42.140515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.140500    2579 scope.go:117] "RemoveContainer" containerID="557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6"
Sep 05 02:30:42.140766 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:30:42.140751    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6\": container with ID starting with 557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6 not found: ID does not exist" containerID="557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6"
Sep 05 02:30:42.140809 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.140768    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6"} err="failed to get container status \"557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6\": rpc error: code = NotFound desc = could not find container \"557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6\": container with ID starting with 557f1ca2db733d9823be7f2a39df5c7a5f32d9a6373c7dde8c454e662b759ea6 not found: ID does not exist"
Sep 05 02:30:42.140809 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.140786    2579 scope.go:117] "RemoveContainer" containerID="972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f"
Sep 05 02:30:42.141023 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:30:42.141006    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f\": container with ID starting with 972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f not found: ID does not exist" containerID="972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f"
Sep 05 02:30:42.141062 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.141027    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f"} err="failed to get container status \"972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f\": rpc error: code = NotFound desc = could not find container \"972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f\": container with ID starting with 972c4a71270e1f50647650f32effc3ab35c9c46a6ea9d554dbca8888a450781f not found: ID does not exist"
Sep 05 02:30:42.550338 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:42.550301    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" path="/var/lib/kubelet/pods/3b6f4cc8-9215-4535-a431-26dd66f663d9/volumes"
Sep 05 02:30:43.092851 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:43.092821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:30:43.093382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:43.093355    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:30:53.093527 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:30:53.093431    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:03.094336 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:03.094241    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:13.093881 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:13.093845    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:23.094279 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:23.094241    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:33.093499 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:33.093398    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:31:42.106824 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.106788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"]
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107184    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy"
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107202    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy"
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107216    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent"
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107224    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent"
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107247    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container"
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107257    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container"
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107270    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="storage-initializer"
Sep 05 02:31:42.107283 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107278    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="storage-initializer"
Sep 05 02:31:42.107630 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107378    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kube-rbac-proxy"
Sep 05 02:31:42.107630 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107388    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="agent"
Sep 05 02:31:42.107630 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.107397    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b6f4cc8-9215-4535-a431-26dd66f663d9" containerName="kserve-container"
Sep 05 02:31:42.110327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.110310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.112481 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.112461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 05 02:31:42.113101 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.113083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 05 02:31:42.121566 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.121526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"]
Sep 05 02:31:42.168650 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.168628    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"]
Sep 05 02:31:42.170265 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.170245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.170330 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.170285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a833f62-a48a-4c06-9cde-f85eae861d76-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.170414 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.170340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtnp7\" (UniqueName: \"kubernetes.io/projected/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-api-access-wtnp7\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.170414 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.170389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.170414 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.170411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.171128 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.171115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.173211 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.173195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 05 02:31:42.183048 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.183027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"]
Sep 05 02:31:42.271253 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cf1335a-e14a-452f-839a-ed9aee14df3c-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.271355 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtnp7\" (UniqueName: \"kubernetes.io/projected/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-api-access-wtnp7\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.271355 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.271470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.271470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n26wg\" (UniqueName: \"kubernetes.io/projected/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-api-access-n26wg\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.271470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.271584 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.271584 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:31:42.271537    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 05 02:31:42.271690 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a833f62-a48a-4c06-9cde-f85eae861d76-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.271690 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:31:42.271606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls podName:4a833f62-a48a-4c06-9cde-f85eae861d76 nodeName:}" failed. No retries permitted until 2026-09-05 02:31:42.77158795 +0000 UTC m=+846.743891821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls") pod "isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" (UID: "4a833f62-a48a-4c06-9cde-f85eae861d76") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 05 02:31:42.271690 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cf1335a-e14a-452f-839a-ed9aee14df3c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.271820 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.271945 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.271929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a833f62-a48a-4c06-9cde-f85eae861d76-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.272026 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.272010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.272075 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.272059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.279633 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.279613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtnp7\" (UniqueName: \"kubernetes.io/projected/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-api-access-wtnp7\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.372522 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.372472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n26wg\" (UniqueName: \"kubernetes.io/projected/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-api-access-n26wg\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.372522 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.372499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.372698 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.372528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cf1335a-e14a-452f-839a-ed9aee14df3c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.372698 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.372549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.372698 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.372570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cf1335a-e14a-452f-839a-ed9aee14df3c-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.372930 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.372908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cf1335a-e14a-452f-839a-ed9aee14df3c-kserve-provision-location\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.373136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.373119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.373196 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.373180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-cabundle-cert\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.374840 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.374823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cf1335a-e14a-452f-839a-ed9aee14df3c-proxy-tls\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.380710 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.380689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n26wg\" (UniqueName: \"kubernetes.io/projected/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-api-access-n26wg\") pod \"isvc-canary-stop-predictor-79f67b4d78-v7v85\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.480488 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.480465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:42.751923 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.751889    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"]
Sep 05 02:31:42.754188 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:31:42.754159    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf1335a_e14a_452f_839a_ed9aee14df3c.slice/crio-7770934cfb10d3d8dcb24b0d549e74e4e14379c7a58aa532ce17fb82de21e5d4 WatchSource:0}: Error finding container 7770934cfb10d3d8dcb24b0d549e74e4e14379c7a58aa532ce17fb82de21e5d4: Status 404 returned error can't find the container with id 7770934cfb10d3d8dcb24b0d549e74e4e14379c7a58aa532ce17fb82de21e5d4
Sep 05 02:31:42.775512 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.775492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:42.777527 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:42.777502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:43.021400 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:43.021374    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:43.094459 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:43.094436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:31:43.290137 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:43.290108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"]
Sep 05 02:31:43.292936 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:31:43.292892    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a833f62_a48a_4c06_9cde_f85eae861d76.slice/crio-0dd522b58188052023dd8d98a3f8e14b1512e138b8d611d12a4dfecafd6819ae WatchSource:0}: Error finding container 0dd522b58188052023dd8d98a3f8e14b1512e138b8d611d12a4dfecafd6819ae: Status 404 returned error can't find the container with id 0dd522b58188052023dd8d98a3f8e14b1512e138b8d611d12a4dfecafd6819ae
Sep 05 02:31:43.403017 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:43.402984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerStarted","Data":"de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941"}
Sep 05 02:31:43.403169 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:43.403063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerStarted","Data":"7770934cfb10d3d8dcb24b0d549e74e4e14379c7a58aa532ce17fb82de21e5d4"}
Sep 05 02:31:43.404741 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:43.404716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerStarted","Data":"e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0"}
Sep 05 02:31:43.404866 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:43.404773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerStarted","Data":"0dd522b58188052023dd8d98a3f8e14b1512e138b8d611d12a4dfecafd6819ae"}
Sep 05 02:31:47.425531 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:47.425497    2579 generic.go:356] "Generic (PLEG): container finished" podID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerID="de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941" exitCode=0
Sep 05 02:31:47.425874 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:47.425568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerDied","Data":"de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941"}
Sep 05 02:31:47.427387 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:47.427363    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerID="e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0" exitCode=0
Sep 05 02:31:47.427509 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:47.427409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerDied","Data":"e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0"}
Sep 05 02:31:48.434244 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.434201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerStarted","Data":"f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc"}
Sep 05 02:31:48.434653 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.434313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerStarted","Data":"944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179"}
Sep 05 02:31:48.434653 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.434535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:48.436496 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.436468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerStarted","Data":"a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43"}
Sep 05 02:31:48.436621 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.436521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerStarted","Data":"61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec"}
Sep 05 02:31:48.436723 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.436705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:48.436800 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.436751    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:48.438382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.438354    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:31:48.455945 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.455900    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podStartSLOduration=2.588100038 podStartE2EDuration="6.455886031s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867785993s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:31:48.453993514 +0000 UTC m=+852.426297392" watchObservedRunningTime="2026-09-05 02:31:48.455886031 +0000 UTC m=+852.428189906"
Sep 05 02:31:48.477079 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:48.477037    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podStartSLOduration=2.7283080440000003 podStartE2EDuration="6.477024143s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.748716099s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:31:48.475776212 +0000 UTC m=+852.448080084" watchObservedRunningTime="2026-09-05 02:31:48.477024143 +0000 UTC m=+852.449328047"
Sep 05 02:31:49.442295 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:49.442255    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:31:49.442692 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:49.442270    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:49.443586 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:49.443559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:31:50.446720 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:50.446651    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:31:54.446933 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:54.446898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:31:54.447412 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:54.447386    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:31:55.451321 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:55.451289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:31:55.451874 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:31:55.451845    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:02.095863 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:02.095830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"]
Sep 05 02:32:02.096294 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:02.096117    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" containerID="cri-o://650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6" gracePeriod=30
Sep 05 02:32:02.096294 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:02.096177    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kube-rbac-proxy" containerID="cri-o://7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228" gracePeriod=30
Sep 05 02:32:02.506416 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:02.506326    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerID="7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228" exitCode=2
Sep 05 02:32:02.506416 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:02.506384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerDied","Data":"7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228"}
Sep 05 02:32:03.087934 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:03.087897    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 05 02:32:03.093870 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:03.093840    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 05 02:32:04.447895 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:04.447856    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:32:05.340564 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.340540    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:32:05.452012 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.451928    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:05.456299 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kserve-provision-location\") pod \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") "
Sep 05 02:32:05.456428 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wh6k\" (UniqueName: \"kubernetes.io/projected/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-api-access-5wh6k\") pod \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") "
Sep 05 02:32:05.456428 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456343    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b08ef4-da4f-47df-8c1b-99981f2b1749-proxy-tls\") pod \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") "
Sep 05 02:32:05.456428 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-rbac-proxy-sar-config\") pod \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") "
Sep 05 02:32:05.456428 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456413    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-cabundle-cert\") pod \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\" (UID: \"d3b08ef4-da4f-47df-8c1b-99981f2b1749\") "
Sep 05 02:32:05.456675 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3b08ef4-da4f-47df-8c1b-99981f2b1749" (UID: "d3b08ef4-da4f-47df-8c1b-99981f2b1749"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:32:05.456813 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3b08ef4-da4f-47df-8c1b-99981f2b1749" (UID: "d3b08ef4-da4f-47df-8c1b-99981f2b1749"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:05.456883 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.456809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3b08ef4-da4f-47df-8c1b-99981f2b1749" (UID: "d3b08ef4-da4f-47df-8c1b-99981f2b1749"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:32:05.458387 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.458365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b08ef4-da4f-47df-8c1b-99981f2b1749-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3b08ef4-da4f-47df-8c1b-99981f2b1749" (UID: "d3b08ef4-da4f-47df-8c1b-99981f2b1749"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:32:05.458576 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.458556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-api-access-5wh6k" (OuterVolumeSpecName: "kube-api-access-5wh6k") pod "d3b08ef4-da4f-47df-8c1b-99981f2b1749" (UID: "d3b08ef4-da4f-47df-8c1b-99981f2b1749"). InnerVolumeSpecName "kube-api-access-5wh6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:32:05.522295 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.522262    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerID="650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6" exitCode=0
Sep 05 02:32:05.522463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.522306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerDied","Data":"650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6"}
Sep 05 02:32:05.522463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.522321    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"
Sep 05 02:32:05.522463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.522355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j" event={"ID":"d3b08ef4-da4f-47df-8c1b-99981f2b1749","Type":"ContainerDied","Data":"498225ba443ef8a86dba0391071498572411cbffb3630a30e4cb6f58c37f565c"}
Sep 05 02:32:05.522463 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.522382    2579 scope.go:117] "RemoveContainer" containerID="7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228"
Sep 05 02:32:05.531233 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.531212    2579 scope.go:117] "RemoveContainer" containerID="650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6"
Sep 05 02:32:05.544152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.544127    2579 scope.go:117] "RemoveContainer" containerID="ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7"
Sep 05 02:32:05.546903 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.546872    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"]
Sep 05 02:32:05.550096 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.550072    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e3535-predictor-778c796d4f-9l44j"]
Sep 05 02:32:05.552591 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.552577    2579 scope.go:117] "RemoveContainer" containerID="7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228"
Sep 05 02:32:05.552891 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:32:05.552872    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228\": container with ID starting with 7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228 not found: ID does not exist" containerID="7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228"
Sep 05 02:32:05.552946 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.552900    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228"} err="failed to get container status \"7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228\": rpc error: code = NotFound desc = could not find container \"7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228\": container with ID starting with 7582d4eb62e9fd2bf50e31bca5b51c986f4a669744ab36c52a8f68b918d3c228 not found: ID does not exist"
Sep 05 02:32:05.552946 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.552920    2579 scope.go:117] "RemoveContainer" containerID="650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6"
Sep 05 02:32:05.553145 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:32:05.553128    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6\": container with ID starting with 650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6 not found: ID does not exist" containerID="650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6"
Sep 05 02:32:05.553180 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.553149    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6"} err="failed to get container status \"650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6\": rpc error: code = NotFound desc = could not find container \"650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6\": container with ID starting with 650a85dbc2ec756fccd556f443ba8f8c55ff3dd36ac4cd9ac91b20bdaa2dd0c6 not found: ID does not exist"
Sep 05 02:32:05.553180 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.553166    2579 scope.go:117] "RemoveContainer" containerID="ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7"
Sep 05 02:32:05.553377 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:32:05.553363    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7\": container with ID starting with ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7 not found: ID does not exist" containerID="ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7"
Sep 05 02:32:05.553412 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.553381    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7"} err="failed to get container status \"ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7\": rpc error: code = NotFound desc = could not find container \"ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7\": container with ID starting with ae349c31f53e0bb7a94eec6e65997742d285900a603f1ddc6f8038ca6e4353b7 not found: ID does not exist"
Sep 05 02:32:05.557813 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.557795    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:32:05.557860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.557816    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wh6k\" (UniqueName: \"kubernetes.io/projected/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-api-access-5wh6k\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:32:05.557860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.557827    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3b08ef4-da4f-47df-8c1b-99981f2b1749-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:32:05.557860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.557838    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:32:05.557860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:05.557846    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3b08ef4-da4f-47df-8c1b-99981f2b1749-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:32:06.556002 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:06.555906    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" path="/var/lib/kubelet/pods/d3b08ef4-da4f-47df-8c1b-99981f2b1749/volumes"
Sep 05 02:32:14.447384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:14.447346    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:32:15.451919 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:15.451881    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:24.447710 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:24.447649    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:32:25.452379 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:25.452337    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:34.448273 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:34.448190    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:32:35.452342 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:35.452300    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:44.448197 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:44.448153    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:32:45.452587 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:45.452541    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:54.448161 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:54.448123    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:32:55.452727 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:55.452686    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:32:58.545733 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:32:58.545691    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 05 02:33:04.448869 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:04.448831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:33:08.555033 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:08.554965    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:33:12.053991 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.053961    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"]
Sep 05 02:33:12.054412 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.054201    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container" containerID="cri-o://944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179" gracePeriod=30
Sep 05 02:33:12.054412 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.054214    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kube-rbac-proxy" containerID="cri-o://f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc" gracePeriod=30
Sep 05 02:33:12.086777 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.086752    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"]
Sep 05 02:33:12.086982 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.086957    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" containerID="cri-o://61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec" gracePeriod=30
Sep 05 02:33:12.087056 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.087006    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kube-rbac-proxy" containerID="cri-o://a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43" gracePeriod=30
Sep 05 02:33:12.855433 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.855402    2579 generic.go:356] "Generic (PLEG): container finished" podID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerID="a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43" exitCode=2
Sep 05 02:33:12.855569 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.855471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerDied","Data":"a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43"}
Sep 05 02:33:12.863407 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.863379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerDied","Data":"f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc"}
Sep 05 02:33:12.863551 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:12.863245    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerID="f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc" exitCode=2
Sep 05 02:33:14.442843 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:14.442801    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 05 02:33:14.448005 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:14.447980    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 05 02:33:15.447214 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:15.447173    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 05 02:33:16.231187 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.231162    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:33:16.249937 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.249916    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:33:16.255735 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.255689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-cabundle-cert\") pod \"4a833f62-a48a-4c06-9cde-f85eae861d76\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") "
Sep 05 02:33:16.255849 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.255734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-rbac-proxy-sar-config\") pod \"4a833f62-a48a-4c06-9cde-f85eae861d76\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") "
Sep 05 02:33:16.255849 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.255754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls\") pod \"4a833f62-a48a-4c06-9cde-f85eae861d76\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") "
Sep 05 02:33:16.255849 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.255810    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtnp7\" (UniqueName: \"kubernetes.io/projected/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-api-access-wtnp7\") pod \"4a833f62-a48a-4c06-9cde-f85eae861d76\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") "
Sep 05 02:33:16.256016 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.255853    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a833f62-a48a-4c06-9cde-f85eae861d76-kserve-provision-location\") pod \"4a833f62-a48a-4c06-9cde-f85eae861d76\" (UID: \"4a833f62-a48a-4c06-9cde-f85eae861d76\") "
Sep 05 02:33:16.256073 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.256017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a833f62-a48a-4c06-9cde-f85eae861d76" (UID: "4a833f62-a48a-4c06-9cde-f85eae861d76"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:16.256119 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.256070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a833f62-a48a-4c06-9cde-f85eae861d76" (UID: "4a833f62-a48a-4c06-9cde-f85eae861d76"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:16.256257 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.256238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a833f62-a48a-4c06-9cde-f85eae861d76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a833f62-a48a-4c06-9cde-f85eae861d76" (UID: "4a833f62-a48a-4c06-9cde-f85eae861d76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:16.257854 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.257830    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a833f62-a48a-4c06-9cde-f85eae861d76" (UID: "4a833f62-a48a-4c06-9cde-f85eae861d76"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:33:16.257937 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.257890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-api-access-wtnp7" (OuterVolumeSpecName: "kube-api-access-wtnp7") pod "4a833f62-a48a-4c06-9cde-f85eae861d76" (UID: "4a833f62-a48a-4c06-9cde-f85eae861d76"). InnerVolumeSpecName "kube-api-access-wtnp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:33:16.357213 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357161    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cf1335a-e14a-452f-839a-ed9aee14df3c-proxy-tls\") pod \"7cf1335a-e14a-452f-839a-ed9aee14df3c\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") "
Sep 05 02:33:16.357335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-cabundle-cert\") pod \"7cf1335a-e14a-452f-839a-ed9aee14df3c\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") "
Sep 05 02:33:16.357335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cf1335a-e14a-452f-839a-ed9aee14df3c-kserve-provision-location\") pod \"7cf1335a-e14a-452f-839a-ed9aee14df3c\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") "
Sep 05 02:33:16.357335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357282    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n26wg\" (UniqueName: \"kubernetes.io/projected/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-api-access-n26wg\") pod \"7cf1335a-e14a-452f-839a-ed9aee14df3c\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") "
Sep 05 02:33:16.357335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-rbac-proxy-sar-config\") pod \"7cf1335a-e14a-452f-839a-ed9aee14df3c\" (UID: \"7cf1335a-e14a-452f-839a-ed9aee14df3c\") "
Sep 05 02:33:16.357553 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357488    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtnp7\" (UniqueName: \"kubernetes.io/projected/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-api-access-wtnp7\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.357553 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357511    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a833f62-a48a-4c06-9cde-f85eae861d76-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.357553 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357527    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.357553 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357541    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a833f62-a48a-4c06-9cde-f85eae861d76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.357767 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357554    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a833f62-a48a-4c06-9cde-f85eae861d76-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.357767 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357579    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cf1335a-e14a-452f-839a-ed9aee14df3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cf1335a-e14a-452f-839a-ed9aee14df3c" (UID: "7cf1335a-e14a-452f-839a-ed9aee14df3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:33:16.357767 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357646    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cf1335a-e14a-452f-839a-ed9aee14df3c" (UID: "7cf1335a-e14a-452f-839a-ed9aee14df3c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:16.357767 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.357683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cf1335a-e14a-452f-839a-ed9aee14df3c" (UID: "7cf1335a-e14a-452f-839a-ed9aee14df3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:33:16.359051 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.359030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf1335a-e14a-452f-839a-ed9aee14df3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cf1335a-e14a-452f-839a-ed9aee14df3c" (UID: "7cf1335a-e14a-452f-839a-ed9aee14df3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:33:16.359190 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.359168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-api-access-n26wg" (OuterVolumeSpecName: "kube-api-access-n26wg") pod "7cf1335a-e14a-452f-839a-ed9aee14df3c" (UID: "7cf1335a-e14a-452f-839a-ed9aee14df3c"). InnerVolumeSpecName "kube-api-access-n26wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:33:16.458534 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.458511    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cf1335a-e14a-452f-839a-ed9aee14df3c-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.458534 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.458531    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.458930 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.458541    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cf1335a-e14a-452f-839a-ed9aee14df3c-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.458930 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.458550    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n26wg\" (UniqueName: \"kubernetes.io/projected/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-api-access-n26wg\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.458930 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.458570    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cf1335a-e14a-452f-839a-ed9aee14df3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:33:16.886803 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.886753    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerID="944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179" exitCode=0
Sep 05 02:33:16.886913 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.886808    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"
Sep 05 02:33:16.886913 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.886834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerDied","Data":"944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179"}
Sep 05 02:33:16.886913 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.886898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5" event={"ID":"4a833f62-a48a-4c06-9cde-f85eae861d76","Type":"ContainerDied","Data":"0dd522b58188052023dd8d98a3f8e14b1512e138b8d611d12a4dfecafd6819ae"}
Sep 05 02:33:16.887076 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.886933    2579 scope.go:117] "RemoveContainer" containerID="f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc"
Sep 05 02:33:16.889225 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.889198    2579 generic.go:356] "Generic (PLEG): container finished" podID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerID="61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec" exitCode=0
Sep 05 02:33:16.889326 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.889256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerDied","Data":"61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec"}
Sep 05 02:33:16.889326 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.889312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85" event={"ID":"7cf1335a-e14a-452f-839a-ed9aee14df3c","Type":"ContainerDied","Data":"7770934cfb10d3d8dcb24b0d549e74e4e14379c7a58aa532ce17fb82de21e5d4"}
Sep 05 02:33:16.889417 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.889260    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"
Sep 05 02:33:16.895648 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.895629    2579 scope.go:117] "RemoveContainer" containerID="944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179"
Sep 05 02:33:16.903593 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.903511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"]
Sep 05 02:33:16.911890 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.903799    2579 scope.go:117] "RemoveContainer" containerID="e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0"
Sep 05 02:33:16.911890 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.909627    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-86f55c65d7-jx9f5"]
Sep 05 02:33:16.912108 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.912054    2579 scope.go:117] "RemoveContainer" containerID="f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc"
Sep 05 02:33:16.912346 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:16.912328    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc\": container with ID starting with f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc not found: ID does not exist" containerID="f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc"
Sep 05 02:33:16.912390 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.912352    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc"} err="failed to get container status \"f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc\": rpc error: code = NotFound desc = could not find container \"f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc\": container with ID starting with f9b8d0ecf4023cb9705985e58087512d229d5307b75e10f411bab26a46696ecc not found: ID does not exist"
Sep 05 02:33:16.912390 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.912368    2579 scope.go:117] "RemoveContainer" containerID="944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179"
Sep 05 02:33:16.912626 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:16.912609    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179\": container with ID starting with 944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179 not found: ID does not exist" containerID="944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179"
Sep 05 02:33:16.912699 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.912631    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179"} err="failed to get container status \"944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179\": rpc error: code = NotFound desc = could not find container \"944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179\": container with ID starting with 944f7e36f2e0e2e0090eb150c0a0b6d1fbf23638f01331086ba8e1d2dd596179 not found: ID does not exist"
Sep 05 02:33:16.912699 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.912644    2579 scope.go:117] "RemoveContainer" containerID="e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0"
Sep 05 02:33:16.912896 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:16.912878    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0\": container with ID starting with e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0 not found: ID does not exist" containerID="e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0"
Sep 05 02:33:16.912942 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.912900    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0"} err="failed to get container status \"e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0\": rpc error: code = NotFound desc = could not find container \"e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0\": container with ID starting with e67e8377b7ee552dbf33717473cb8553d50b175504f5d33ae00afd9197cdc6b0 not found: ID does not exist"
Sep 05 02:33:16.912942 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.912914    2579 scope.go:117] "RemoveContainer" containerID="a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43"
Sep 05 02:33:16.920087 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.920070    2579 scope.go:117] "RemoveContainer" containerID="61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec"
Sep 05 02:33:16.923400 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.923380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"]
Sep 05 02:33:16.927697 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.927675    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-79f67b4d78-v7v85"]
Sep 05 02:33:16.928512 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.928491    2579 scope.go:117] "RemoveContainer" containerID="de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941"
Sep 05 02:33:16.935314 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.935300    2579 scope.go:117] "RemoveContainer" containerID="a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43"
Sep 05 02:33:16.935570 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:16.935553    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43\": container with ID starting with a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43 not found: ID does not exist" containerID="a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43"
Sep 05 02:33:16.935633 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.935576    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43"} err="failed to get container status \"a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43\": rpc error: code = NotFound desc = could not find container \"a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43\": container with ID starting with a7be6acef7a1f7bc8742129111092bbed75fb556f95694f64de0e0d8058dec43 not found: ID does not exist"
Sep 05 02:33:16.935633 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.935591    2579 scope.go:117] "RemoveContainer" containerID="61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec"
Sep 05 02:33:16.935851 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:16.935832    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec\": container with ID starting with 61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec not found: ID does not exist" containerID="61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec"
Sep 05 02:33:16.935892 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.935856    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec"} err="failed to get container status \"61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec\": rpc error: code = NotFound desc = could not find container \"61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec\": container with ID starting with 61b1adf67466986ad4c11c59f4238055f913b967dda888437e7c81a6704f24ec not found: ID does not exist"
Sep 05 02:33:16.935892 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.935868    2579 scope.go:117] "RemoveContainer" containerID="de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941"
Sep 05 02:33:16.936090 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:16.936072    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941\": container with ID starting with de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941 not found: ID does not exist" containerID="de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941"
Sep 05 02:33:16.936128 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:16.936096    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941"} err="failed to get container status \"de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941\": rpc error: code = NotFound desc = could not find container \"de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941\": container with ID starting with de8ec8af7c294954bd0b6aaca666526f676884dca2c0e095837b71ab0ff8f941 not found: ID does not exist"
Sep 05 02:33:18.551208 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:18.551167    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" path="/var/lib/kubelet/pods/4a833f62-a48a-4c06-9cde-f85eae861d76/volumes"
Sep 05 02:33:18.551719 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:18.551700    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" path="/var/lib/kubelet/pods/7cf1335a-e14a-452f-839a-ed9aee14df3c/volumes"
Sep 05 02:33:22.357610 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357580    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"]
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357906    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357918    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357938    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="storage-initializer"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357944    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="storage-initializer"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357952    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kube-rbac-proxy"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357958    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kube-rbac-proxy"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357965    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kube-rbac-proxy"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357970    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kube-rbac-proxy"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357977    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357983    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container"
Sep 05 02:33:22.357984 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357990    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kube-rbac-proxy"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.357995    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kube-rbac-proxy"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358002    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="storage-initializer"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358007    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="storage-initializer"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358012    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="storage-initializer"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358017    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="storage-initializer"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358023    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358028    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358077    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kube-rbac-proxy"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358084    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kserve-container"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358090    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kube-rbac-proxy"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358095    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cf1335a-e14a-452f-839a-ed9aee14df3c" containerName="kserve-container"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358102    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a833f62-a48a-4c06-9cde-f85eae861d76" containerName="kserve-container"
Sep 05 02:33:22.358504 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.358109    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3b08ef4-da4f-47df-8c1b-99981f2b1749" containerName="kube-rbac-proxy"
Sep 05 02:33:22.362941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.362923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.369549 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.369381    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b3e077-kube-rbac-proxy-sar-config\""
Sep 05 02:33:22.369549 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.369385    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:33:22.369774 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.369753    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b3e077-predictor-serving-cert\""
Sep 05 02:33:22.369882 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.369856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:33:22.370157 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.370138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:33:22.370652 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.370640    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:33:22.370830 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.370817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:33:22.383587 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.383565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"]
Sep 05 02:33:22.398062 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.398037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.398171 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.398066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-proxy-tls\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.398171 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.398085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kserve-provision-location\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.398171 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.398148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5pd\" (UniqueName: \"kubernetes.io/projected/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-api-access-wn5pd\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.398340 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.398212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-cabundle-cert\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.498919 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.498893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-cabundle-cert\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.499026 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.498933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.499026 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.498952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-proxy-tls\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.499026 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.498969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kserve-provision-location\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.499172 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.499079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5pd\" (UniqueName: \"kubernetes.io/projected/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-api-access-wn5pd\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.499314 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.499295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kserve-provision-location\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.499576 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.499559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.499702 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.499656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-cabundle-cert\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.501470 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.501454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-proxy-tls\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.506905 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.506880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5pd\" (UniqueName: \"kubernetes.io/projected/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-api-access-wn5pd\") pod \"isvc-primary-b3e077-predictor-54d44d9966-kjzpf\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") " pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.673149 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.673094    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:22.946395 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:22.946369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"]
Sep 05 02:33:22.949292 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:33:22.949264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3d320f_9e4e_4ff6_bc8e_d2c6e4d04450.slice/crio-ac71fe1f35a25ee7a982bf35bc5e2ecaedc9ba6d327bcd5c7a4f1ebead250610 WatchSource:0}: Error finding container ac71fe1f35a25ee7a982bf35bc5e2ecaedc9ba6d327bcd5c7a4f1ebead250610: Status 404 returned error can't find the container with id ac71fe1f35a25ee7a982bf35bc5e2ecaedc9ba6d327bcd5c7a4f1ebead250610
Sep 05 02:33:23.935055 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:23.935012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerStarted","Data":"a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3"}
Sep 05 02:33:23.935517 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:23.935105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerStarted","Data":"ac71fe1f35a25ee7a982bf35bc5e2ecaedc9ba6d327bcd5c7a4f1ebead250610"}
Sep 05 02:33:26.950234 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:26.950202    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerID="a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3" exitCode=0
Sep 05 02:33:26.950561 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:26.950274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerDied","Data":"a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3"}
Sep 05 02:33:27.957832 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:27.957797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerStarted","Data":"fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76"}
Sep 05 02:33:27.958191 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:27.957865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerStarted","Data":"987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a"}
Sep 05 02:33:27.958191 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:27.958089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:27.978413 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:27.978369    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podStartSLOduration=2.205935659 podStartE2EDuration="5.978355252s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.772419593s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:33:27.976453197 +0000 UTC m=+951.948757115" watchObservedRunningTime="2026-09-05 02:33:27.978355252 +0000 UTC m=+951.950659128"
Sep 05 02:33:28.962166 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:28.962142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:28.963335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:28.963308    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:33:29.966505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:29.966464    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:33:34.970516 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:34.970489    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:33:34.970995 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:34.970945    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:33:44.971628 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:44.971505    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:33:52.419959 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.419922    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"]
Sep 05 02:33:52.427273 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.427223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.430138 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.430102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"]
Sep 05 02:33:52.430429 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.430411    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-d98d3-predictor-serving-cert\""
Sep 05 02:33:52.430513 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.430432    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-d98d3-kube-rbac-proxy-sar-config\""
Sep 05 02:33:52.536754 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.536719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4461d229-4797-4421-8b28-4752425fe60b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.536858 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.536780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.536905 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.536872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbwr\" (UniqueName: \"kubernetes.io/projected/4461d229-4797-4421-8b28-4752425fe60b-kube-api-access-nsbwr\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.637453 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.637428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.637587 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.637480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbwr\" (UniqueName: \"kubernetes.io/projected/4461d229-4797-4421-8b28-4752425fe60b-kube-api-access-nsbwr\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.637587 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.637517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4461d229-4797-4421-8b28-4752425fe60b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.637733 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:52.637603    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-serving-cert: secret "message-dumper-raw-d98d3-predictor-serving-cert" not found
Sep 05 02:33:52.637733 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:33:52.637723    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls podName:4461d229-4797-4421-8b28-4752425fe60b nodeName:}" failed. No retries permitted until 2026-09-05 02:33:53.13770254 +0000 UTC m=+977.110006411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls") pod "message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" (UID: "4461d229-4797-4421-8b28-4752425fe60b") : secret "message-dumper-raw-d98d3-predictor-serving-cert" not found
Sep 05 02:33:52.638136 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.638117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4461d229-4797-4421-8b28-4752425fe60b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:52.647434 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:52.647417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbwr\" (UniqueName: \"kubernetes.io/projected/4461d229-4797-4421-8b28-4752425fe60b-kube-api-access-nsbwr\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:53.141303 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:53.141272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:53.143511 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:53.143491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls\") pod \"message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") " pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:53.338654 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:53.338625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:53.607537 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:53.607514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"]
Sep 05 02:33:53.609300 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:33:53.609272    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4461d229_4797_4421_8b28_4752425fe60b.slice/crio-5a9fde46c04f2f1ed5f5658d13e1a40260388e09109c9485d607547ae622cc0f WatchSource:0}: Error finding container 5a9fde46c04f2f1ed5f5658d13e1a40260388e09109c9485d607547ae622cc0f: Status 404 returned error can't find the container with id 5a9fde46c04f2f1ed5f5658d13e1a40260388e09109c9485d607547ae622cc0f
Sep 05 02:33:53.611584 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:53.611565    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 02:33:54.078680 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:54.078632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" event={"ID":"4461d229-4797-4421-8b28-4752425fe60b","Type":"ContainerStarted","Data":"5a9fde46c04f2f1ed5f5658d13e1a40260388e09109c9485d607547ae622cc0f"}
Sep 05 02:33:54.971132 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:54.971093    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:33:55.086033 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:55.085939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" event={"ID":"4461d229-4797-4421-8b28-4752425fe60b","Type":"ContainerStarted","Data":"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"}
Sep 05 02:33:55.086033 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:55.086013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" event={"ID":"4461d229-4797-4421-8b28-4752425fe60b","Type":"ContainerStarted","Data":"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"}
Sep 05 02:33:55.086236 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:55.086048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:55.107903 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:55.107853    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" podStartSLOduration=1.951185637 podStartE2EDuration="3.107834323s" totalImagesPullingTime="1.156648686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:33:55.10654594 +0000 UTC m=+979.078849816" watchObservedRunningTime="2026-09-05 02:33:55.107834323 +0000 UTC m=+979.080138266"
Sep 05 02:33:56.090844 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:56.090819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:33:56.092715 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:33:56.092692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:34:03.106760 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:34:03.106683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:34:04.971369 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:34:04.971329    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:34:14.972435 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:34:14.972318    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:34:24.971137 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:34:24.971097    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:34:34.971131 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:34:34.971087    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 05 02:34:44.971710 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:34:44.971657    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:35:25.561400 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:25.561368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"]
Sep 05 02:35:25.562037 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:25.561678    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container" containerID="cri-o://987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a" gracePeriod=30
Sep 05 02:35:25.562037 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:25.561737    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kube-rbac-proxy" containerID="cri-o://fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76" gracePeriod=30
Sep 05 02:35:26.537437 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:26.537403    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerID="fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76" exitCode=2
Sep 05 02:35:26.537601 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:26.537489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerDied","Data":"fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76"}
Sep 05 02:35:29.406127 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.406103    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:35:29.524545 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.524521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-cabundle-cert\") pod \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") "
Sep 05 02:35:29.524700 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.524578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-rbac-proxy-sar-config\") pod \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") "
Sep 05 02:35:29.524700 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.524649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-proxy-tls\") pod \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") "
Sep 05 02:35:29.524823 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.524696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kserve-provision-location\") pod \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") "
Sep 05 02:35:29.524823 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.524739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn5pd\" (UniqueName: \"kubernetes.io/projected/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-api-access-wn5pd\") pod \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\" (UID: \"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450\") "
Sep 05 02:35:29.525017 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.524893    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" (UID: "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:29.525068 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.525004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" (UID: "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:29.525068 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.525010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" (UID: "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:29.526646 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.526626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" (UID: "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:29.526753 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.526691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-api-access-wn5pd" (OuterVolumeSpecName: "kube-api-access-wn5pd") pod "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" (UID: "5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450"). InnerVolumeSpecName "kube-api-access-wn5pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:29.552523 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.552493    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerID="987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a" exitCode=0
Sep 05 02:35:29.552609 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.552547    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"
Sep 05 02:35:29.552609 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.552579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerDied","Data":"987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a"}
Sep 05 02:35:29.552741 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.552628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf" event={"ID":"5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450","Type":"ContainerDied","Data":"ac71fe1f35a25ee7a982bf35bc5e2ecaedc9ba6d327bcd5c7a4f1ebead250610"}
Sep 05 02:35:29.552741 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.552653    2579 scope.go:117] "RemoveContainer" containerID="fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76"
Sep 05 02:35:29.561396 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.561377    2579 scope.go:117] "RemoveContainer" containerID="987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a"
Sep 05 02:35:29.568515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.568500    2579 scope.go:117] "RemoveContainer" containerID="a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3"
Sep 05 02:35:29.575256 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.575229    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"]
Sep 05 02:35:29.576903 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.576888    2579 scope.go:117] "RemoveContainer" containerID="fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76"
Sep 05 02:35:29.577168 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:29.577146    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76\": container with ID starting with fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76 not found: ID does not exist" containerID="fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76"
Sep 05 02:35:29.577242 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.577175    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76"} err="failed to get container status \"fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76\": rpc error: code = NotFound desc = could not find container \"fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76\": container with ID starting with fe5831f9eb734668b832ecb4b9a85154d785d9a55bac9861db9d5fa8a6595c76 not found: ID does not exist"
Sep 05 02:35:29.577242 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.577194    2579 scope.go:117] "RemoveContainer" containerID="987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a"
Sep 05 02:35:29.577477 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:29.577451    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a\": container with ID starting with 987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a not found: ID does not exist" containerID="987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a"
Sep 05 02:35:29.577538 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.577481    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a"} err="failed to get container status \"987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a\": rpc error: code = NotFound desc = could not find container \"987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a\": container with ID starting with 987c7abb0b3fd0e090aeb98eb1f81887456fe2dc770090a02e9705bdcf62334a not found: ID does not exist"
Sep 05 02:35:29.577538 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.577497    2579 scope.go:117] "RemoveContainer" containerID="a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3"
Sep 05 02:35:29.577754 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:29.577737    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3\": container with ID starting with a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3 not found: ID does not exist" containerID="a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3"
Sep 05 02:35:29.577824 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.577758    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3"} err="failed to get container status \"a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3\": rpc error: code = NotFound desc = could not find container \"a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3\": container with ID starting with a348aa8f0a35fd5cc77455d05e8190fe0151ad8b9a9ea7eb108d5ae8d233b1f3 not found: ID does not exist"
Sep 05 02:35:29.578167 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.578149    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b3e077-predictor-54d44d9966-kjzpf"]
Sep 05 02:35:29.626382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.626334    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:29.626382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.626373    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:29.626382 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.626388    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5pd\" (UniqueName: \"kubernetes.io/projected/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-api-access-wn5pd\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:29.626625 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.626401    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:29.626625 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:29.626414    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:30.557068 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.553319    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" path="/var/lib/kubelet/pods/5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450/volumes"
Sep 05 02:35:30.729864 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.729831    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"]
Sep 05 02:35:30.730157 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730144    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container"
Sep 05 02:35:30.730209 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730158    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container"
Sep 05 02:35:30.730209 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730174    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="storage-initializer"
Sep 05 02:35:30.730209 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730179    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="storage-initializer"
Sep 05 02:35:30.730209 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730188    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kube-rbac-proxy"
Sep 05 02:35:30.730209 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730193    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kube-rbac-proxy"
Sep 05 02:35:30.730385 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730248    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kserve-container"
Sep 05 02:35:30.730385 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.730256    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3d320f-9e4e-4ff6-bc8e-d2c6e4d04450" containerName="kube-rbac-proxy"
Sep 05 02:35:30.734747 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.734725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.737254 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.737229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:35:30.737341 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.737231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4bf2b4-kube-rbac-proxy-sar-config\""
Sep 05 02:35:30.737341 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.737305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-4bf2b4-dockercfg-gw8hf\""
Sep 05 02:35:30.737454 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.737407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-4bf2b4\""
Sep 05 02:35:30.737921 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.737903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4bf2b4-predictor-serving-cert\""
Sep 05 02:35:30.743014 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.742994    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"]
Sep 05 02:35:30.836425 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.836370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngxt\" (UniqueName: \"kubernetes.io/projected/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-api-access-wngxt\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.836425 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.836404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-cabundle-cert\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.836585 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.836437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.836585 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.836517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.836585 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.836553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a015b6-421b-4ae8-83b2-584a6a1db608-kserve-provision-location\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.937839 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.937812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a015b6-421b-4ae8-83b2-584a6a1db608-kserve-provision-location\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.937923 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.937858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngxt\" (UniqueName: \"kubernetes.io/projected/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-api-access-wngxt\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.937923 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.937894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-cabundle-cert\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.937998 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.937932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.938128 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.938071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.938231 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:30.938171    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-serving-cert: secret "isvc-init-fail-4bf2b4-predictor-serving-cert" not found
Sep 05 02:35:30.938231 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:30.938227    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls podName:75a015b6-421b-4ae8-83b2-584a6a1db608 nodeName:}" failed. No retries permitted until 2026-09-05 02:35:31.438211898 +0000 UTC m=+1075.410515756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls") pod "isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" (UID: "75a015b6-421b-4ae8-83b2-584a6a1db608") : secret "isvc-init-fail-4bf2b4-predictor-serving-cert" not found
Sep 05 02:35:30.938321 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.938230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a015b6-421b-4ae8-83b2-584a6a1db608-kserve-provision-location\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.938503 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.938486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.938556 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.938538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-cabundle-cert\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:30.947652 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:30.947624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngxt\" (UniqueName: \"kubernetes.io/projected/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-api-access-wngxt\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:31.442411 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:31.442357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:31.444894 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:31.444868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls\") pod \"isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") " pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:31.646112 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:31.646077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:31.918654 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:31.918624    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"]
Sep 05 02:35:31.920550 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:35:31.920522    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a015b6_421b_4ae8_83b2_584a6a1db608.slice/crio-aa045e407c6f84f182df039d967fef8b086ffa811ae3c8e7cd3d6c7a258c27d1 WatchSource:0}: Error finding container aa045e407c6f84f182df039d967fef8b086ffa811ae3c8e7cd3d6c7a258c27d1: Status 404 returned error can't find the container with id aa045e407c6f84f182df039d967fef8b086ffa811ae3c8e7cd3d6c7a258c27d1
Sep 05 02:35:32.572075 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:32.572037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" event={"ID":"75a015b6-421b-4ae8-83b2-584a6a1db608","Type":"ContainerStarted","Data":"0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42"}
Sep 05 02:35:32.572280 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:32.572102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" event={"ID":"75a015b6-421b-4ae8-83b2-584a6a1db608","Type":"ContainerStarted","Data":"aa045e407c6f84f182df039d967fef8b086ffa811ae3c8e7cd3d6c7a258c27d1"}
Sep 05 02:35:37.594544 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:37.594515    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74_75a015b6-421b-4ae8-83b2-584a6a1db608/storage-initializer/0.log"
Sep 05 02:35:37.594938 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:37.594552    2579 generic.go:356] "Generic (PLEG): container finished" podID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerID="0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42" exitCode=1
Sep 05 02:35:37.594938 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:37.594630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" event={"ID":"75a015b6-421b-4ae8-83b2-584a6a1db608","Type":"ContainerDied","Data":"0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42"}
Sep 05 02:35:38.605247 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:38.605178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74_75a015b6-421b-4ae8-83b2-584a6a1db608/storage-initializer/0.log"
Sep 05 02:35:38.606144 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:38.605344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" event={"ID":"75a015b6-421b-4ae8-83b2-584a6a1db608","Type":"ContainerStarted","Data":"b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac"}
Sep 05 02:35:40.717296 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:40.717215    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"]
Sep 05 02:35:40.718263 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:40.717739    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerName="storage-initializer" containerID="cri-o://b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac" gracePeriod=30
Sep 05 02:35:42.370768 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.370746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74_75a015b6-421b-4ae8-83b2-584a6a1db608/storage-initializer/1.log"
Sep 05 02:35:42.371131 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.371117    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74_75a015b6-421b-4ae8-83b2-584a6a1db608/storage-initializer/0.log"
Sep 05 02:35:42.373473 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.373450    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:42.445077 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445019    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wngxt\" (UniqueName: \"kubernetes.io/projected/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-api-access-wngxt\") pod \"75a015b6-421b-4ae8-83b2-584a6a1db608\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") "
Sep 05 02:35:42.445077 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445055    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-cabundle-cert\") pod \"75a015b6-421b-4ae8-83b2-584a6a1db608\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") "
Sep 05 02:35:42.445220 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445124    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-rbac-proxy-sar-config\") pod \"75a015b6-421b-4ae8-83b2-584a6a1db608\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") "
Sep 05 02:35:42.445220 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a015b6-421b-4ae8-83b2-584a6a1db608-kserve-provision-location\") pod \"75a015b6-421b-4ae8-83b2-584a6a1db608\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") "
Sep 05 02:35:42.445220 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445186    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls\") pod \"75a015b6-421b-4ae8-83b2-584a6a1db608\" (UID: \"75a015b6-421b-4ae8-83b2-584a6a1db608\") "
Sep 05 02:35:42.445456 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a015b6-421b-4ae8-83b2-584a6a1db608-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75a015b6-421b-4ae8-83b2-584a6a1db608" (UID: "75a015b6-421b-4ae8-83b2-584a6a1db608"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:35:42.445588 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "75a015b6-421b-4ae8-83b2-584a6a1db608" (UID: "75a015b6-421b-4ae8-83b2-584a6a1db608"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:42.445588 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.445513    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75a015b6-421b-4ae8-83b2-584a6a1db608" (UID: "75a015b6-421b-4ae8-83b2-584a6a1db608"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:42.447166 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.447141    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-api-access-wngxt" (OuterVolumeSpecName: "kube-api-access-wngxt") pod "75a015b6-421b-4ae8-83b2-584a6a1db608" (UID: "75a015b6-421b-4ae8-83b2-584a6a1db608"). InnerVolumeSpecName "kube-api-access-wngxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:42.447266 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.447241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75a015b6-421b-4ae8-83b2-584a6a1db608" (UID: "75a015b6-421b-4ae8-83b2-584a6a1db608"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:42.545723 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.545698    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngxt\" (UniqueName: \"kubernetes.io/projected/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-api-access-wngxt\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:42.545723 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.545722    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:42.545860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.545738    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75a015b6-421b-4ae8-83b2-584a6a1db608-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:42.545860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.545751    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75a015b6-421b-4ae8-83b2-584a6a1db608-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:42.545860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.545765    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75a015b6-421b-4ae8-83b2-584a6a1db608-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:42.629304 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.629286    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74_75a015b6-421b-4ae8-83b2-584a6a1db608/storage-initializer/1.log"
Sep 05 02:35:42.629746 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.629711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74_75a015b6-421b-4ae8-83b2-584a6a1db608/storage-initializer/0.log"
Sep 05 02:35:42.629814 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.629768    2579 generic.go:356] "Generic (PLEG): container finished" podID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerID="b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac" exitCode=1
Sep 05 02:35:42.629814 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.629796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" event={"ID":"75a015b6-421b-4ae8-83b2-584a6a1db608","Type":"ContainerDied","Data":"b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac"}
Sep 05 02:35:42.629896 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.629819    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"
Sep 05 02:35:42.629896 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.629854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74" event={"ID":"75a015b6-421b-4ae8-83b2-584a6a1db608","Type":"ContainerDied","Data":"aa045e407c6f84f182df039d967fef8b086ffa811ae3c8e7cd3d6c7a258c27d1"}
Sep 05 02:35:42.629896 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.629885    2579 scope.go:117] "RemoveContainer" containerID="b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac"
Sep 05 02:35:42.638200 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.638184    2579 scope.go:117] "RemoveContainer" containerID="0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42"
Sep 05 02:35:42.645591 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.645574    2579 scope.go:117] "RemoveContainer" containerID="b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac"
Sep 05 02:35:42.645883 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:42.645862    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac\": container with ID starting with b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac not found: ID does not exist" containerID="b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac"
Sep 05 02:35:42.645933 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.645893    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac"} err="failed to get container status \"b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac\": rpc error: code = NotFound desc = could not find container \"b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac\": container with ID starting with b51be3a22691051067a873b9fdc1d6e21e16b346708ce14c08c3f8bb684c64ac not found: ID does not exist"
Sep 05 02:35:42.645933 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.645915    2579 scope.go:117] "RemoveContainer" containerID="0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42"
Sep 05 02:35:42.646159 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:42.646141    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42\": container with ID starting with 0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42 not found: ID does not exist" containerID="0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42"
Sep 05 02:35:42.646208 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.646165    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42"} err="failed to get container status \"0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42\": rpc error: code = NotFound desc = could not find container \"0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42\": container with ID starting with 0fd97319153b65b620689350360350838b5bbcb71d1550341b2f598f476d9e42 not found: ID does not exist"
Sep 05 02:35:42.659427 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.659405    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"]
Sep 05 02:35:42.665912 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:42.665889    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4bf2b4-predictor-6b9d5bf6bc-xng74"]
Sep 05 02:35:44.552086 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:44.552046    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" path="/var/lib/kubelet/pods/75a015b6-421b-4ae8-83b2-584a6a1db608/volumes"
Sep 05 02:35:45.946237 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.946201    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"]
Sep 05 02:35:45.946572 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.946532    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerName="storage-initializer"
Sep 05 02:35:45.946572 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.946543    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerName="storage-initializer"
Sep 05 02:35:45.946572 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.946564    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerName="storage-initializer"
Sep 05 02:35:45.946572 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.946569    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerName="storage-initializer"
Sep 05 02:35:45.946738 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.946623    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerName="storage-initializer"
Sep 05 02:35:45.946738 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.946631    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a015b6-421b-4ae8-83b2-584a6a1db608" containerName="storage-initializer"
Sep 05 02:35:45.951350 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.951329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:45.953583 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.953564    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-03136-predictor-serving-cert\""
Sep 05 02:35:45.953689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.953619    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:35:45.953689 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.953572    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-03136-kube-rbac-proxy-sar-config\""
Sep 05 02:35:45.953780 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.953567    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:35:45.962077 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:45.962057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"]
Sep 05 02:35:46.073475 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.073444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7zt9\" (UniqueName: \"kubernetes.io/projected/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-api-access-q7zt9\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.073582 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.073482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35df0da2-be00-4a20-b516-3c9f82bc8934-kserve-provision-location\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.073582 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.073508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.073738 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.073603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-cabundle-cert\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.073738 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.073657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35df0da2-be00-4a20-b516-3c9f82bc8934-proxy-tls\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.174809 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.174786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-cabundle-cert\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.174922 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.174820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35df0da2-be00-4a20-b516-3c9f82bc8934-proxy-tls\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.174922 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.174856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7zt9\" (UniqueName: \"kubernetes.io/projected/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-api-access-q7zt9\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.174922 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.174875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35df0da2-be00-4a20-b516-3c9f82bc8934-kserve-provision-location\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.174922 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.174907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.175351 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.175331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35df0da2-be00-4a20-b516-3c9f82bc8934-kserve-provision-location\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.175614 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.175589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-cabundle-cert\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.175687 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.175595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.177370 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.177345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35df0da2-be00-4a20-b516-3c9f82bc8934-proxy-tls\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.182987 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.182957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7zt9\" (UniqueName: \"kubernetes.io/projected/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-api-access-q7zt9\") pod \"raw-sklearn-03136-predictor-84fcd88955-hs42x\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") " pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.261589 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.261568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:46.538132 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.538054    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"]
Sep 05 02:35:46.541191 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:35:46.541161    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35df0da2_be00_4a20_b516_3c9f82bc8934.slice/crio-cdf04c01562fdb774411ce12d7509bbefd97ec2efbff3d1910ef9e425e9b6119 WatchSource:0}: Error finding container cdf04c01562fdb774411ce12d7509bbefd97ec2efbff3d1910ef9e425e9b6119: Status 404 returned error can't find the container with id cdf04c01562fdb774411ce12d7509bbefd97ec2efbff3d1910ef9e425e9b6119
Sep 05 02:35:46.650587 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.650558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerStarted","Data":"3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef"}
Sep 05 02:35:46.650738 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:46.650616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerStarted","Data":"cdf04c01562fdb774411ce12d7509bbefd97ec2efbff3d1910ef9e425e9b6119"}
Sep 05 02:35:47.461643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.461608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7_4461d229-4797-4421-8b28-4752425fe60b/kserve-container/0.log"
Sep 05 02:35:47.636281 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.636240    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"]
Sep 05 02:35:47.636488 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.636434    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kserve-container" containerID="cri-o://5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a" gracePeriod=30
Sep 05 02:35:47.636552 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.636523    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kube-rbac-proxy" containerID="cri-o://790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0" gracePeriod=30
Sep 05 02:35:47.888518 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.888491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:35:47.989778 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.989749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4461d229-4797-4421-8b28-4752425fe60b-kube-rbac-proxy-sar-config\") pod \"4461d229-4797-4421-8b28-4752425fe60b\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") "
Sep 05 02:35:47.989898 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.989827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls\") pod \"4461d229-4797-4421-8b28-4752425fe60b\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") "
Sep 05 02:35:47.989898 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.989864    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsbwr\" (UniqueName: \"kubernetes.io/projected/4461d229-4797-4421-8b28-4752425fe60b-kube-api-access-nsbwr\") pod \"4461d229-4797-4421-8b28-4752425fe60b\" (UID: \"4461d229-4797-4421-8b28-4752425fe60b\") "
Sep 05 02:35:47.990155 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.990131    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4461d229-4797-4421-8b28-4752425fe60b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4461d229-4797-4421-8b28-4752425fe60b" (UID: "4461d229-4797-4421-8b28-4752425fe60b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:35:47.991839 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.991819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4461d229-4797-4421-8b28-4752425fe60b" (UID: "4461d229-4797-4421-8b28-4752425fe60b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:35:47.991936 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:47.991914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4461d229-4797-4421-8b28-4752425fe60b-kube-api-access-nsbwr" (OuterVolumeSpecName: "kube-api-access-nsbwr") pod "4461d229-4797-4421-8b28-4752425fe60b" (UID: "4461d229-4797-4421-8b28-4752425fe60b"). InnerVolumeSpecName "kube-api-access-nsbwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:35:48.090823 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.090787    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsbwr\" (UniqueName: \"kubernetes.io/projected/4461d229-4797-4421-8b28-4752425fe60b-kube-api-access-nsbwr\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:48.090823 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.090818    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4461d229-4797-4421-8b28-4752425fe60b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:48.090823 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.090828    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4461d229-4797-4421-8b28-4752425fe60b-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:35:48.665788 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.665755    2579 generic.go:356] "Generic (PLEG): container finished" podID="4461d229-4797-4421-8b28-4752425fe60b" containerID="790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0" exitCode=2
Sep 05 02:35:48.665788 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.665782    2579 generic.go:356] "Generic (PLEG): container finished" podID="4461d229-4797-4421-8b28-4752425fe60b" containerID="5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a" exitCode=2
Sep 05 02:35:48.666232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.665820    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"
Sep 05 02:35:48.666232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.665829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" event={"ID":"4461d229-4797-4421-8b28-4752425fe60b","Type":"ContainerDied","Data":"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"}
Sep 05 02:35:48.666232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.665875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" event={"ID":"4461d229-4797-4421-8b28-4752425fe60b","Type":"ContainerDied","Data":"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"}
Sep 05 02:35:48.666232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.665895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7" event={"ID":"4461d229-4797-4421-8b28-4752425fe60b","Type":"ContainerDied","Data":"5a9fde46c04f2f1ed5f5658d13e1a40260388e09109c9485d607547ae622cc0f"}
Sep 05 02:35:48.666232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.665919    2579 scope.go:117] "RemoveContainer" containerID="790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"
Sep 05 02:35:48.674685 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.674645    2579 scope.go:117] "RemoveContainer" containerID="5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"
Sep 05 02:35:48.682893 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.682875    2579 scope.go:117] "RemoveContainer" containerID="790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"
Sep 05 02:35:48.683219 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:48.683201    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0\": container with ID starting with 790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0 not found: ID does not exist" containerID="790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"
Sep 05 02:35:48.683333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.683225    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"} err="failed to get container status \"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0\": rpc error: code = NotFound desc = could not find container \"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0\": container with ID starting with 790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0 not found: ID does not exist"
Sep 05 02:35:48.683333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.683242    2579 scope.go:117] "RemoveContainer" containerID="5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"
Sep 05 02:35:48.683525 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:35:48.683503    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a\": container with ID starting with 5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a not found: ID does not exist" containerID="5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"
Sep 05 02:35:48.683588 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.683535    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"} err="failed to get container status \"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a\": rpc error: code = NotFound desc = could not find container \"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a\": container with ID starting with 5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a not found: ID does not exist"
Sep 05 02:35:48.683588 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.683558    2579 scope.go:117] "RemoveContainer" containerID="790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"
Sep 05 02:35:48.683852 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.683833    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0"} err="failed to get container status \"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0\": rpc error: code = NotFound desc = could not find container \"790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0\": container with ID starting with 790ac4b332488440797b1d0609703ee439f83b5ca05337e462df8230a4e9a9f0 not found: ID does not exist"
Sep 05 02:35:48.683852 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.683850    2579 scope.go:117] "RemoveContainer" containerID="5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"
Sep 05 02:35:48.684070 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.684049    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"]
Sep 05 02:35:48.684111 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.684081    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a"} err="failed to get container status \"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a\": rpc error: code = NotFound desc = could not find container \"5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a\": container with ID starting with 5aa62cc984c7773f97c62dfa53f07889e6b7f276b0bee47fd4d2c89e56eb577a not found: ID does not exist"
Sep 05 02:35:48.690137 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:48.690117    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d98d3-predictor-5887996fd4-tfzz7"]
Sep 05 02:35:50.552475 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:50.552452    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4461d229-4797-4421-8b28-4752425fe60b" path="/var/lib/kubelet/pods/4461d229-4797-4421-8b28-4752425fe60b/volumes"
Sep 05 02:35:50.679014 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:50.678989    2579 generic.go:356] "Generic (PLEG): container finished" podID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerID="3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef" exitCode=0
Sep 05 02:35:50.679111 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:50.679019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerDied","Data":"3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef"}
Sep 05 02:35:51.686221 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:51.686186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerStarted","Data":"cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821"}
Sep 05 02:35:51.686588 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:51.686276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerStarted","Data":"205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706"}
Sep 05 02:35:51.686588 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:51.686479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:51.708865 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:51.708817    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podStartSLOduration=2.763718049 podStartE2EDuration="6.708803014s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.945084965s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:35:51.706222533 +0000 UTC m=+1095.678526408" watchObservedRunningTime="2026-09-05 02:35:51.708803014 +0000 UTC m=+1095.681106890"
Sep 05 02:35:52.691383 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:52.691351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:52.692712 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:52.692678    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:35:53.696210 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:53.696163    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:35:58.701232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:58.701199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:35:58.701886 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:35:58.701651    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:36:08.702129 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:36:08.702084    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:36:18.701634 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:36:18.701594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:36:28.702188 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:36:28.702143    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:36:38.702072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:36:38.702028    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:36:48.702072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:36:48.702029    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:36:58.701784 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:36:58.701744    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:37:08.702873 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:08.702832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:37:16.010422 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:16.010388    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"]
Sep 05 02:37:16.010920 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:16.010628    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" containerID="cri-o://205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706" gracePeriod=30
Sep 05 02:37:16.010920 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:16.010691    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kube-rbac-proxy" containerID="cri-o://cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821" gracePeriod=30
Sep 05 02:37:17.104737 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:17.104690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerDied","Data":"cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821"}
Sep 05 02:37:17.105143 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:17.104686    2579 generic.go:356] "Generic (PLEG): container finished" podID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerID="cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821" exitCode=2
Sep 05 02:37:18.696682 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:18.696622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 05 02:37:18.702641 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:18.702598    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 05 02:37:19.764273 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.764252    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:37:19.875372 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35df0da2-be00-4a20-b516-3c9f82bc8934-proxy-tls\") pod \"35df0da2-be00-4a20-b516-3c9f82bc8934\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") "
Sep 05 02:37:19.875372 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-rbac-proxy-sar-config\") pod \"35df0da2-be00-4a20-b516-3c9f82bc8934\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") "
Sep 05 02:37:19.875539 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35df0da2-be00-4a20-b516-3c9f82bc8934-kserve-provision-location\") pod \"35df0da2-be00-4a20-b516-3c9f82bc8934\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") "
Sep 05 02:37:19.875539 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7zt9\" (UniqueName: \"kubernetes.io/projected/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-api-access-q7zt9\") pod \"35df0da2-be00-4a20-b516-3c9f82bc8934\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") "
Sep 05 02:37:19.875539 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-cabundle-cert\") pod \"35df0da2-be00-4a20-b516-3c9f82bc8934\" (UID: \"35df0da2-be00-4a20-b516-3c9f82bc8934\") "
Sep 05 02:37:19.875730 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35df0da2-be00-4a20-b516-3c9f82bc8934" (UID: "35df0da2-be00-4a20-b516-3c9f82bc8934"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:37:19.875808 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35df0da2-be00-4a20-b516-3c9f82bc8934-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35df0da2-be00-4a20-b516-3c9f82bc8934" (UID: "35df0da2-be00-4a20-b516-3c9f82bc8934"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:37:19.875864 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.875841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35df0da2-be00-4a20-b516-3c9f82bc8934" (UID: "35df0da2-be00-4a20-b516-3c9f82bc8934"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:37:19.877363 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.877343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35df0da2-be00-4a20-b516-3c9f82bc8934-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35df0da2-be00-4a20-b516-3c9f82bc8934" (UID: "35df0da2-be00-4a20-b516-3c9f82bc8934"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:37:19.877412 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.877368    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-api-access-q7zt9" (OuterVolumeSpecName: "kube-api-access-q7zt9") pod "35df0da2-be00-4a20-b516-3c9f82bc8934" (UID: "35df0da2-be00-4a20-b516-3c9f82bc8934"). InnerVolumeSpecName "kube-api-access-q7zt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:37:19.976185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.976161    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35df0da2-be00-4a20-b516-3c9f82bc8934-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:37:19.976185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.976182    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:37:19.976301 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.976192    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35df0da2-be00-4a20-b516-3c9f82bc8934-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:37:19.976301 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.976200    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7zt9\" (UniqueName: \"kubernetes.io/projected/35df0da2-be00-4a20-b516-3c9f82bc8934-kube-api-access-q7zt9\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:37:19.976301 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:19.976209    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35df0da2-be00-4a20-b516-3c9f82bc8934-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:37:20.121232 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.121209    2579 generic.go:356] "Generic (PLEG): container finished" podID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerID="205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706" exitCode=0
Sep 05 02:37:20.121338 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.121263    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"
Sep 05 02:37:20.121338 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.121289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerDied","Data":"205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706"}
Sep 05 02:37:20.121460 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.121357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x" event={"ID":"35df0da2-be00-4a20-b516-3c9f82bc8934","Type":"ContainerDied","Data":"cdf04c01562fdb774411ce12d7509bbefd97ec2efbff3d1910ef9e425e9b6119"}
Sep 05 02:37:20.121460 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.121402    2579 scope.go:117] "RemoveContainer" containerID="cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821"
Sep 05 02:37:20.129603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.129588    2579 scope.go:117] "RemoveContainer" containerID="205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706"
Sep 05 02:37:20.136604 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.136587    2579 scope.go:117] "RemoveContainer" containerID="3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef"
Sep 05 02:37:20.141857 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.141838    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"]
Sep 05 02:37:20.144311 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.144295    2579 scope.go:117] "RemoveContainer" containerID="cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821"
Sep 05 02:37:20.144573 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:37:20.144555    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821\": container with ID starting with cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821 not found: ID does not exist" containerID="cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821"
Sep 05 02:37:20.144618 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.144580    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821"} err="failed to get container status \"cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821\": rpc error: code = NotFound desc = could not find container \"cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821\": container with ID starting with cd160d3e548e865ca7703f523f38c6f2be8ce535ba089e7580dd243a275d8821 not found: ID does not exist"
Sep 05 02:37:20.144618 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.144595    2579 scope.go:117] "RemoveContainer" containerID="205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706"
Sep 05 02:37:20.144863 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:37:20.144846    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706\": container with ID starting with 205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706 not found: ID does not exist" containerID="205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706"
Sep 05 02:37:20.144910 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.144869    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706"} err="failed to get container status \"205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706\": rpc error: code = NotFound desc = could not find container \"205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706\": container with ID starting with 205f62cf4eb473abba0ec13d40751ad93104c13cd16fe3e798e147c89e8f3706 not found: ID does not exist"
Sep 05 02:37:20.144910 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.144884    2579 scope.go:117] "RemoveContainer" containerID="3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef"
Sep 05 02:37:20.145139 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:37:20.145118    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef\": container with ID starting with 3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef not found: ID does not exist" containerID="3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef"
Sep 05 02:37:20.145185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.145147    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef"} err="failed to get container status \"3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef\": rpc error: code = NotFound desc = could not find container \"3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef\": container with ID starting with 3cafa6191224c8c715c5d6fcb88868933d9bb168085abc917a37d8aaa1c19bef not found: ID does not exist"
Sep 05 02:37:20.145446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.145425    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-03136-predictor-84fcd88955-hs42x"]
Sep 05 02:37:20.550971 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:20.550941    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" path="/var/lib/kubelet/pods/35df0da2-be00-4a20-b516-3c9f82bc8934/volumes"
Sep 05 02:37:21.215227 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215197    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"]
Sep 05 02:37:21.215540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215506    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kube-rbac-proxy"
Sep 05 02:37:21.215540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215516    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kube-rbac-proxy"
Sep 05 02:37:21.215540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215531    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kserve-container"
Sep 05 02:37:21.215540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215536    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kserve-container"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215551    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215557    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215568    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kube-rbac-proxy"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215573    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kube-rbac-proxy"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215579    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="storage-initializer"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215584    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="storage-initializer"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215633    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kube-rbac-proxy"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215642    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4461d229-4797-4421-8b28-4752425fe60b" containerName="kserve-container"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215648    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kserve-container"
Sep 05 02:37:21.215693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.215653    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35df0da2-be00-4a20-b516-3c9f82bc8934" containerName="kube-rbac-proxy"
Sep 05 02:37:21.220302 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.220283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.222427 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.222406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:37:21.222515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.222413    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:37:21.222515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.222460    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-17b5f-kube-rbac-proxy-sar-config\""
Sep 05 02:37:21.222515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.222471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-17b5f-predictor-serving-cert\""
Sep 05 02:37:21.222790 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.222773    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:37:21.223483 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.223465    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:37:21.223657 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.223641    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:37:21.228601 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.228581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"]
Sep 05 02:37:21.387707 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.387676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kserve-provision-location\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.387810 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.387738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-cabundle-cert\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.387810 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.387764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwp8q\" (UniqueName: \"kubernetes.io/projected/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-api-access-jwp8q\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.387810 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.387795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eafa449-7095-4e63-b4b9-186ecc3ef98d-proxy-tls\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.387915 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.387812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.488761 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.488705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-cabundle-cert\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.488761 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.488745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwp8q\" (UniqueName: \"kubernetes.io/projected/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-api-access-jwp8q\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.488875 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.488778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eafa449-7095-4e63-b4b9-186ecc3ef98d-proxy-tls\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.488875 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.488795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.488875 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.488813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kserve-provision-location\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.489203 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.489183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kserve-provision-location\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.489446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.489415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-cabundle-cert\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.489446 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.489437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.491188 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.491170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eafa449-7095-4e63-b4b9-186ecc3ef98d-proxy-tls\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.497229 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.497204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwp8q\" (UniqueName: \"kubernetes.io/projected/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-api-access-jwp8q\") pod \"raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.531280 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.531256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:21.803339 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:21.803307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"]
Sep 05 02:37:21.806292 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:37:21.806267    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eafa449_7095_4e63_b4b9_186ecc3ef98d.slice/crio-6f50473081785dfd69fa47770235e004e81d8e3b54ed35edc170cd30496567eb WatchSource:0}: Error finding container 6f50473081785dfd69fa47770235e004e81d8e3b54ed35edc170cd30496567eb: Status 404 returned error can't find the container with id 6f50473081785dfd69fa47770235e004e81d8e3b54ed35edc170cd30496567eb
Sep 05 02:37:22.134238 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:22.134150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerStarted","Data":"e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8"}
Sep 05 02:37:22.134238 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:22.134212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerStarted","Data":"6f50473081785dfd69fa47770235e004e81d8e3b54ed35edc170cd30496567eb"}
Sep 05 02:37:26.156215 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:26.156177    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerID="e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8" exitCode=0
Sep 05 02:37:26.156568 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:26.156245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerDied","Data":"e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8"}
Sep 05 02:37:27.169003 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:27.168966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerStarted","Data":"028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0"}
Sep 05 02:37:27.169360 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:27.169047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerStarted","Data":"f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2"}
Sep 05 02:37:27.169360 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:27.169240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:27.190829 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:27.190789    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podStartSLOduration=2.307471474 podStartE2EDuration="6.190776654s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88330518s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:37:27.188880934 +0000 UTC m=+1191.161184811" watchObservedRunningTime="2026-09-05 02:37:27.190776654 +0000 UTC m=+1191.163080530"
Sep 05 02:37:28.173163 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:28.173130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:28.174468 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:28.174434    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:37:29.177363 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:29.177322    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:37:34.181457 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:34.181414    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:37:34.182175 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:34.182134    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:37:44.182857 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:44.182820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:37:54.182293 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:37:54.182248    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:38:04.182957 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:04.182919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:38:14.182399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:14.182353    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:38:24.182682 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:24.182623    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:38:34.182483 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:34.182390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:38:43.098783 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.098752    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"]
Sep 05 02:38:43.102360 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.102338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.104822 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.104801    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 05 02:38:43.104942 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.104803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 05 02:38:43.119722 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.119699    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"]
Sep 05 02:38:43.165319 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.165295    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"]
Sep 05 02:38:43.172077 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.172054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.176197 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.176176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 05 02:38:43.186497 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.186479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"]
Sep 05 02:38:43.206589 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.206567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.206703 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.206609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfn8\" (UniqueName: \"kubernetes.io/projected/1397e7db-2881-4a73-89e9-c61361df9b07-kube-api-access-lbfn8\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.206703 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.206630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1397e7db-2881-4a73-89e9-c61361df9b07-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.206703 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.206654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.206836 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.206707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.307830 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.307806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.307921 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.307836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.307921 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.307862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfn8\" (UniqueName: \"kubernetes.io/projected/1397e7db-2881-4a73-89e9-c61361df9b07-kube-api-access-lbfn8\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.307921 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.307898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1397e7db-2881-4a73-89e9-c61361df9b07-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.308030 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.307925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.308030 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.307950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.308101 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.308071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.308139 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.308111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.308202 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.308166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887hj\" (UniqueName: \"kubernetes.io/projected/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-api-access-887hj\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.308247 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.308230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.308291 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.308278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1397e7db-2881-4a73-89e9-c61361df9b07-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.308328 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:38:43.308314    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 05 02:38:43.308396 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:38:43.308385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls podName:1397e7db-2881-4a73-89e9-c61361df9b07 nodeName:}" failed. No retries permitted until 2026-09-05 02:38:43.808369527 +0000 UTC m=+1267.780673382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls") pod "isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" (UID: "1397e7db-2881-4a73-89e9-c61361df9b07") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 05 02:38:43.308697 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.308649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.308747 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.308656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.320728 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.320702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfn8\" (UniqueName: \"kubernetes.io/projected/1397e7db-2881-4a73-89e9-c61361df9b07-kube-api-access-lbfn8\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.408527 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.408473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.408527 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.408506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.408655 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.408530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887hj\" (UniqueName: \"kubernetes.io/projected/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-api-access-887hj\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.408655 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.408578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.408655 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.408596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.408950 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.408928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kserve-provision-location\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.409208 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.409188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-cabundle-cert\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.409304 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.409282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.410860 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.410839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-proxy-tls\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.418534 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.418508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-887hj\" (UniqueName: \"kubernetes.io/projected/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-api-access-887hj\") pod \"isvc-canary-create-predictor-54dc6cb966-dsmct\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.481273 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.481250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:43.765630 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.765603    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"]
Sep 05 02:38:43.766800 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:38:43.766773    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cd880b_e8dc_42d6_9235_d6548c2f9c6b.slice/crio-38140ea8ba127b6e5942f11c725325d744d7b5c7889d13bfc05f8e3f11d6037a WatchSource:0}: Error finding container 38140ea8ba127b6e5942f11c725325d744d7b5c7889d13bfc05f8e3f11d6037a: Status 404 returned error can't find the container with id 38140ea8ba127b6e5942f11c725325d744d7b5c7889d13bfc05f8e3f11d6037a
Sep 05 02:38:43.811588 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.811566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:43.813799 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:43.813778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:44.012177 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:44.012151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:44.182838 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:44.182805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:38:44.287756 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:44.287724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"]
Sep 05 02:38:44.290257 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:38:44.290234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1397e7db_2881_4a73_89e9_c61361df9b07.slice/crio-f6b10bc4726eafade37edd8c98588cd409d9bb6ce111d5a3ec9133373402ed5c WatchSource:0}: Error finding container f6b10bc4726eafade37edd8c98588cd409d9bb6ce111d5a3ec9133373402ed5c: Status 404 returned error can't find the container with id f6b10bc4726eafade37edd8c98588cd409d9bb6ce111d5a3ec9133373402ed5c
Sep 05 02:38:44.550904 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:44.550831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerStarted","Data":"0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a"}
Sep 05 02:38:44.550904 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:44.550887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerStarted","Data":"f6b10bc4726eafade37edd8c98588cd409d9bb6ce111d5a3ec9133373402ed5c"}
Sep 05 02:38:44.551072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:44.550908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerStarted","Data":"db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119"}
Sep 05 02:38:44.551072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:44.550930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerStarted","Data":"38140ea8ba127b6e5942f11c725325d744d7b5c7889d13bfc05f8e3f11d6037a"}
Sep 05 02:38:48.569141 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:48.569106    2579 generic.go:356] "Generic (PLEG): container finished" podID="1397e7db-2881-4a73-89e9-c61361df9b07" containerID="0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a" exitCode=0
Sep 05 02:38:48.569615 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:48.569184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerDied","Data":"0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a"}
Sep 05 02:38:48.571092 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:48.571068    2579 generic.go:356] "Generic (PLEG): container finished" podID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerID="db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119" exitCode=0
Sep 05 02:38:48.571188 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:48.571145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerDied","Data":"db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119"}
Sep 05 02:38:49.579784 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.579735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerStarted","Data":"3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57"}
Sep 05 02:38:49.580245 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.579858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerStarted","Data":"10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662"}
Sep 05 02:38:49.580245 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.580148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:49.582152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.582121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerStarted","Data":"d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73"}
Sep 05 02:38:49.582264 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.582183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerStarted","Data":"efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3"}
Sep 05 02:38:49.582438 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.582419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:49.582514 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.582467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:49.584252 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.584220    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:38:49.603642 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.603589    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podStartSLOduration=3.014069052 podStartE2EDuration="6.603570829s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.589501777s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:38:49.602035846 +0000 UTC m=+1273.574339724" watchObservedRunningTime="2026-09-05 02:38:49.603570829 +0000 UTC m=+1273.575874786"
Sep 05 02:38:49.624430 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:49.624371    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podStartSLOduration=2.869291754 podStartE2EDuration="6.624352912s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.755061158s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:38:49.623253794 +0000 UTC m=+1273.595557671" watchObservedRunningTime="2026-09-05 02:38:49.624352912 +0000 UTC m=+1273.596656788"
Sep 05 02:38:50.587185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:50.587141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:38:50.587603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:50.587216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:50.588842 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:50.588814    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:38:51.278295 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:51.278259    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"]
Sep 05 02:38:51.278591 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:51.278542    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" containerID="cri-o://f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2" gracePeriod=30
Sep 05 02:38:51.278687 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:51.278572    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kube-rbac-proxy" containerID="cri-o://028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0" gracePeriod=30
Sep 05 02:38:51.593315 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:51.593246    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerID="028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0" exitCode=2
Sep 05 02:38:51.593641 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:51.593316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerDied","Data":"028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0"}
Sep 05 02:38:51.593641 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:51.593580    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:38:54.178550 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:54.178501    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 05 02:38:54.182868 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:54.182829    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 05 02:38:55.595337 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:55.595269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:38:55.596345 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:55.596277    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:38:56.603187 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:56.603116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:38:56.604204 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:56.604126    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:38:58.121634 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.121613    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:38:58.243715 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.243640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-rbac-proxy-sar-config\") pod \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") "
Sep 05 02:38:58.243715 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.243708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eafa449-7095-4e63-b4b9-186ecc3ef98d-proxy-tls\") pod \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") "
Sep 05 02:38:58.243906 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.243749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kserve-provision-location\") pod \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") "
Sep 05 02:38:58.243906 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.243791    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-cabundle-cert\") pod \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") "
Sep 05 02:38:58.243906 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.243816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwp8q\" (UniqueName: \"kubernetes.io/projected/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-api-access-jwp8q\") pod \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\" (UID: \"6eafa449-7095-4e63-b4b9-186ecc3ef98d\") "
Sep 05 02:38:58.244062 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.243992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6eafa449-7095-4e63-b4b9-186ecc3ef98d" (UID: "6eafa449-7095-4e63-b4b9-186ecc3ef98d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:58.244116 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.244088    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6eafa449-7095-4e63-b4b9-186ecc3ef98d" (UID: "6eafa449-7095-4e63-b4b9-186ecc3ef98d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:38:58.244152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.244117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6eafa449-7095-4e63-b4b9-186ecc3ef98d" (UID: "6eafa449-7095-4e63-b4b9-186ecc3ef98d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:38:58.245771 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.245747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eafa449-7095-4e63-b4b9-186ecc3ef98d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6eafa449-7095-4e63-b4b9-186ecc3ef98d" (UID: "6eafa449-7095-4e63-b4b9-186ecc3ef98d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:38:58.245855 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.245842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-api-access-jwp8q" (OuterVolumeSpecName: "kube-api-access-jwp8q") pod "6eafa449-7095-4e63-b4b9-186ecc3ef98d" (UID: "6eafa449-7095-4e63-b4b9-186ecc3ef98d"). InnerVolumeSpecName "kube-api-access-jwp8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:38:58.344399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.344377    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:38:58.344399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.344398    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eafa449-7095-4e63-b4b9-186ecc3ef98d-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:38:58.344540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.344408    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:38:58.344540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.344417    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eafa449-7095-4e63-b4b9-186ecc3ef98d-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:38:58.344540 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.344425    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwp8q\" (UniqueName: \"kubernetes.io/projected/6eafa449-7095-4e63-b4b9-186ecc3ef98d-kube-api-access-jwp8q\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:38:58.639279 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.639247    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerID="f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2" exitCode=0
Sep 05 02:38:58.639404 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.639311    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"
Sep 05 02:38:58.639404 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.639336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerDied","Data":"f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2"}
Sep 05 02:38:58.639519 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.639408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68" event={"ID":"6eafa449-7095-4e63-b4b9-186ecc3ef98d","Type":"ContainerDied","Data":"6f50473081785dfd69fa47770235e004e81d8e3b54ed35edc170cd30496567eb"}
Sep 05 02:38:58.639519 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.639449    2579 scope.go:117] "RemoveContainer" containerID="028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0"
Sep 05 02:38:58.647649 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.647630    2579 scope.go:117] "RemoveContainer" containerID="f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2"
Sep 05 02:38:58.654950 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.654932    2579 scope.go:117] "RemoveContainer" containerID="e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8"
Sep 05 02:38:58.657696 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.657677    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"]
Sep 05 02:38:58.661311 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.661292    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-17b5f-predictor-7bb86784c-sbp68"]
Sep 05 02:38:58.663357 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.663341    2579 scope.go:117] "RemoveContainer" containerID="028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0"
Sep 05 02:38:58.663609 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:38:58.663592    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0\": container with ID starting with 028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0 not found: ID does not exist" containerID="028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0"
Sep 05 02:38:58.663671 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.663617    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0"} err="failed to get container status \"028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0\": rpc error: code = NotFound desc = could not find container \"028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0\": container with ID starting with 028505e7eaf6c4d0494e6eddb0c705c5d2cc3d371bd01f4a4b445eddb8d34cf0 not found: ID does not exist"
Sep 05 02:38:58.663671 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.663634    2579 scope.go:117] "RemoveContainer" containerID="f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2"
Sep 05 02:38:58.663901 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:38:58.663883    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2\": container with ID starting with f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2 not found: ID does not exist" containerID="f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2"
Sep 05 02:38:58.663949 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.663906    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2"} err="failed to get container status \"f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2\": rpc error: code = NotFound desc = could not find container \"f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2\": container with ID starting with f9d72d1485559bf1365e4a5d8abe6ecae3f8a1d10fdcb103626be2b3f34ae3c2 not found: ID does not exist"
Sep 05 02:38:58.663949 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.663921    2579 scope.go:117] "RemoveContainer" containerID="e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8"
Sep 05 02:38:58.664163 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:38:58.664145    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8\": container with ID starting with e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8 not found: ID does not exist" containerID="e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8"
Sep 05 02:38:58.664205 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:38:58.664169    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8"} err="failed to get container status \"e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8\": rpc error: code = NotFound desc = could not find container \"e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8\": container with ID starting with e4db63dc35d094107ba03e96623a73647c2d8b6d8ee3723be0c6da88e9684fb8 not found: ID does not exist"
Sep 05 02:39:00.550941 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:00.550907    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" path="/var/lib/kubelet/pods/6eafa449-7095-4e63-b4b9-186ecc3ef98d/volumes"
Sep 05 02:39:05.596475 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:05.596431    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:39:06.603747 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:06.603704    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:39:15.596322 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:15.596271    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:39:16.604182 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:16.604141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:39:25.595937 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:25.595894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:39:26.604028 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:26.603988    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:39:35.596375 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:35.596335    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:39:36.604352 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:36.604304    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:39:45.596746 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:45.596580    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:39:46.605004 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:46.604880    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:39:55.596268 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:55.596224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:39:56.603929 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:39:56.603868    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:40:04.551624 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:04.551554    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:40:05.546263 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:05.546234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:40:13.139318 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.139274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"]
Sep 05 02:40:13.139788 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.139560    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" containerID="cri-o://efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3" gracePeriod=30
Sep 05 02:40:13.139788 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.139598    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kube-rbac-proxy" containerID="cri-o://d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73" gracePeriod=30
Sep 05 02:40:13.150971 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.150939    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"]
Sep 05 02:40:13.151244 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.151213    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" containerID="cri-o://10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662" gracePeriod=30
Sep 05 02:40:13.151332 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.151280    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kube-rbac-proxy" containerID="cri-o://3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57" gracePeriod=30
Sep 05 02:40:13.995098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.995061    2579 generic.go:356] "Generic (PLEG): container finished" podID="1397e7db-2881-4a73-89e9-c61361df9b07" containerID="3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57" exitCode=2
Sep 05 02:40:13.995275 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.995133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerDied","Data":"3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57"}
Sep 05 02:40:13.997099 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.997081    2579 generic.go:356] "Generic (PLEG): container finished" podID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerID="d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73" exitCode=2
Sep 05 02:40:13.997209 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:13.997148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerDied","Data":"d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73"}
Sep 05 02:40:14.545705 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:14.545641    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 05 02:40:15.546626 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:15.546585    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 05 02:40:15.587535 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:15.587494    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 05 02:40:16.594213 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:16.594170    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 05 02:40:17.790968 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.790939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:40:17.894243 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894218    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-887hj\" (UniqueName: \"kubernetes.io/projected/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-api-access-887hj\") pod \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") "
Sep 05 02:40:17.894368 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-proxy-tls\") pod \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") "
Sep 05 02:40:17.894368 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-cabundle-cert\") pod \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") "
Sep 05 02:40:17.894368 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kserve-provision-location\") pod \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") "
Sep 05 02:40:17.894535 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894475    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-rbac-proxy-sar-config\") pod \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\" (UID: \"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b\") "
Sep 05 02:40:17.894676 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" (UID: "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:17.894799 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" (UID: "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:17.894905 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894830    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" (UID: "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:17.894905 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.894838    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:17.896265 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.896239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-api-access-887hj" (OuterVolumeSpecName: "kube-api-access-887hj") pod "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" (UID: "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b"). InnerVolumeSpecName "kube-api-access-887hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:17.896417 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.896401    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" (UID: "e4cd880b-e8dc-42d6-9235-d6548c2f9c6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:17.902001 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.901983    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:40:17.995565 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.995544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls\") pod \"1397e7db-2881-4a73-89e9-c61361df9b07\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") "
Sep 05 02:40:17.995705 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.995572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-kube-rbac-proxy-sar-config\") pod \"1397e7db-2881-4a73-89e9-c61361df9b07\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") "
Sep 05 02:40:17.995705 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.995603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1397e7db-2881-4a73-89e9-c61361df9b07-kserve-provision-location\") pod \"1397e7db-2881-4a73-89e9-c61361df9b07\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") "
Sep 05 02:40:17.995826 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.995787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbfn8\" (UniqueName: \"kubernetes.io/projected/1397e7db-2881-4a73-89e9-c61361df9b07-kube-api-access-lbfn8\") pod \"1397e7db-2881-4a73-89e9-c61361df9b07\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") "
Sep 05 02:40:17.995826 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.995820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-cabundle-cert\") pod \"1397e7db-2881-4a73-89e9-c61361df9b07\" (UID: \"1397e7db-2881-4a73-89e9-c61361df9b07\") "
Sep 05 02:40:17.995956 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.995882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1397e7db-2881-4a73-89e9-c61361df9b07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1397e7db-2881-4a73-89e9-c61361df9b07" (UID: "1397e7db-2881-4a73-89e9-c61361df9b07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:40:17.995956 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.995933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1397e7db-2881-4a73-89e9-c61361df9b07" (UID: "1397e7db-2881-4a73-89e9-c61361df9b07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:17.996241 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.996215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1397e7db-2881-4a73-89e9-c61361df9b07" (UID: "1397e7db-2881-4a73-89e9-c61361df9b07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:40:17.996241 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.996233    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887hj\" (UniqueName: \"kubernetes.io/projected/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-api-access-887hj\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:17.996377 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.996251    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:17.996377 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.996267    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:17.996377 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.996275    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:17.996377 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.996283    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1397e7db-2881-4a73-89e9-c61361df9b07-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:17.996377 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.996292    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:17.997711 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.997688    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1397e7db-2881-4a73-89e9-c61361df9b07-kube-api-access-lbfn8" (OuterVolumeSpecName: "kube-api-access-lbfn8") pod "1397e7db-2881-4a73-89e9-c61361df9b07" (UID: "1397e7db-2881-4a73-89e9-c61361df9b07"). InnerVolumeSpecName "kube-api-access-lbfn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:40:17.997786 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:17.997739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1397e7db-2881-4a73-89e9-c61361df9b07" (UID: "1397e7db-2881-4a73-89e9-c61361df9b07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:40:18.018457 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.018433    2579 generic.go:356] "Generic (PLEG): container finished" podID="1397e7db-2881-4a73-89e9-c61361df9b07" containerID="10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662" exitCode=0
Sep 05 02:40:18.018556 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.018487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"
Sep 05 02:40:18.018556 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.018497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerDied","Data":"10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662"}
Sep 05 02:40:18.018696 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.018559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl" event={"ID":"1397e7db-2881-4a73-89e9-c61361df9b07","Type":"ContainerDied","Data":"f6b10bc4726eafade37edd8c98588cd409d9bb6ce111d5a3ec9133373402ed5c"}
Sep 05 02:40:18.018696 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.018594    2579 scope.go:117] "RemoveContainer" containerID="3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57"
Sep 05 02:40:18.020804 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.020778    2579 generic.go:356] "Generic (PLEG): container finished" podID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerID="efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3" exitCode=0
Sep 05 02:40:18.020903 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.020815    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"
Sep 05 02:40:18.020903 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.020847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerDied","Data":"efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3"}
Sep 05 02:40:18.020997 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.020905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct" event={"ID":"e4cd880b-e8dc-42d6-9235-d6548c2f9c6b","Type":"ContainerDied","Data":"38140ea8ba127b6e5942f11c725325d744d7b5c7889d13bfc05f8e3f11d6037a"}
Sep 05 02:40:18.028426 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.028406    2579 scope.go:117] "RemoveContainer" containerID="10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662"
Sep 05 02:40:18.041524 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.041504    2579 scope.go:117] "RemoveContainer" containerID="0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a"
Sep 05 02:40:18.047825 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.047793    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"]
Sep 05 02:40:18.049162 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.049148    2579 scope.go:117] "RemoveContainer" containerID="3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57"
Sep 05 02:40:18.049434 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.049417    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57\": container with ID starting with 3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57 not found: ID does not exist" containerID="3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57"
Sep 05 02:40:18.049508 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.049440    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57"} err="failed to get container status \"3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57\": rpc error: code = NotFound desc = could not find container \"3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57\": container with ID starting with 3b03a5bb48e3b8b0349c1fc10a145dafa4c298d64578e998ce47903efeb18b57 not found: ID does not exist"
Sep 05 02:40:18.049508 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.049464    2579 scope.go:117] "RemoveContainer" containerID="10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662"
Sep 05 02:40:18.049716 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.049699    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662\": container with ID starting with 10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662 not found: ID does not exist" containerID="10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662"
Sep 05 02:40:18.049763 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.049722    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662"} err="failed to get container status \"10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662\": rpc error: code = NotFound desc = could not find container \"10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662\": container with ID starting with 10a6311f885dc6d0eb20644b6a79e8f56f7e7b6cdabaa0b95cf6566e6ba8c662 not found: ID does not exist"
Sep 05 02:40:18.049763 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.049737    2579 scope.go:117] "RemoveContainer" containerID="0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a"
Sep 05 02:40:18.049989 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.049973    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a\": container with ID starting with 0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a not found: ID does not exist" containerID="0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a"
Sep 05 02:40:18.050026 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.049994    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a"} err="failed to get container status \"0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a\": rpc error: code = NotFound desc = could not find container \"0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a\": container with ID starting with 0fab281823196266209e74e87ffeb84a9788991c30f57baaaad6c6d783de760a not found: ID does not exist"
Sep 05 02:40:18.050026 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.050007    2579 scope.go:117] "RemoveContainer" containerID="d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73"
Sep 05 02:40:18.055996 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.055976    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6f8d77bbc9-h48kl"]
Sep 05 02:40:18.057617 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.057591    2579 scope.go:117] "RemoveContainer" containerID="efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3"
Sep 05 02:40:18.064585 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.064563    2579 scope.go:117] "RemoveContainer" containerID="db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119"
Sep 05 02:40:18.066125 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.066105    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"]
Sep 05 02:40:18.068122 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.068104    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-54dc6cb966-dsmct"]
Sep 05 02:40:18.071808 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.071792    2579 scope.go:117] "RemoveContainer" containerID="d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73"
Sep 05 02:40:18.072032 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.072015    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73\": container with ID starting with d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73 not found: ID does not exist" containerID="d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73"
Sep 05 02:40:18.072075 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.072038    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73"} err="failed to get container status \"d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73\": rpc error: code = NotFound desc = could not find container \"d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73\": container with ID starting with d94319faa835b0b99a1c04d16d7731bfbe6141a6045d346756d78dcd146a9b73 not found: ID does not exist"
Sep 05 02:40:18.072075 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.072052    2579 scope.go:117] "RemoveContainer" containerID="efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3"
Sep 05 02:40:18.072257 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.072240    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3\": container with ID starting with efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3 not found: ID does not exist" containerID="efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3"
Sep 05 02:40:18.072297 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.072261    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3"} err="failed to get container status \"efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3\": rpc error: code = NotFound desc = could not find container \"efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3\": container with ID starting with efd6de4c66a96e6a5ef357cd548dddeb78ef5c8f1fcb618028d98f266e3ce3d3 not found: ID does not exist"
Sep 05 02:40:18.072297 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.072273    2579 scope.go:117] "RemoveContainer" containerID="db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119"
Sep 05 02:40:18.072480 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.072467    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119\": container with ID starting with db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119 not found: ID does not exist" containerID="db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119"
Sep 05 02:40:18.072560 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.072483    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119"} err="failed to get container status \"db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119\": rpc error: code = NotFound desc = could not find container \"db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119\": container with ID starting with db1512f7bd1fec4742ce11069ae4a87eecc373451afee82a262fdf1cc2978119 not found: ID does not exist"
Sep 05 02:40:18.096955 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.096934    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfn8\" (UniqueName: \"kubernetes.io/projected/1397e7db-2881-4a73-89e9-c61361df9b07-kube-api-access-lbfn8\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:18.096955 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.096954    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1397e7db-2881-4a73-89e9-c61361df9b07-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:18.097061 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.096963    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1397e7db-2881-4a73-89e9-c61361df9b07-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:40:18.285982 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.285957    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"]
Sep 05 02:40:18.286355 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286338    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286358    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286382    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286391    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286402    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="storage-initializer"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286410    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="storage-initializer"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286420    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286428    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286438    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container"
Sep 05 02:40:18.286445 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286446    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286458    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286466    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286478    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="storage-initializer"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286486    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="storage-initializer"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286505    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286512    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286527    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="storage-initializer"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286536    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="storage-initializer"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286620    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kserve-container"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286633    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kserve-container"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286644    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286653    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eafa449-7095-4e63-b4b9-186ecc3ef98d" containerName="kube-rbac-proxy"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286685    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" containerName="kserve-container"
Sep 05 02:40:18.286911 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.286694    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" containerName="kube-rbac-proxy"
Sep 05 02:40:18.291693 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.291654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.293861 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.293840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:40:18.293979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.293847    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 05 02:40:18.293979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.293883    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 05 02:40:18.293979 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.293940    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:40:18.294240 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.294220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:40:18.294327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.294318    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 05 02:40:18.294388 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.294343    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 05 02:40:18.303744 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.303725    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"]
Sep 05 02:40:18.354947 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.354921    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"]
Sep 05 02:40:18.358297 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.358281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.360281 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.360264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 05 02:40:18.368924 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.368905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"]
Sep 05 02:40:18.400000 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.399979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.400098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdzf\" (UniqueName: \"kubernetes.io/projected/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-api-access-2fdzf\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.400098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15397d25-488e-4ba5-8363-ea07ebe908ce-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.400189 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.400189 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9264867-1aec-45e7-9435-685c2e411772-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.400189 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.400313 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.400313 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rf8j\" (UniqueName: \"kubernetes.io/projected/e9264867-1aec-45e7-9435-685c2e411772-kube-api-access-9rf8j\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.400313 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.400313 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.400260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9264867-1aec-45e7-9435-685c2e411772-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.501135 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.501224 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdzf\" (UniqueName: \"kubernetes.io/projected/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-api-access-2fdzf\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.501224 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15397d25-488e-4ba5-8363-ea07ebe908ce-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.501224 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.501384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9264867-1aec-45e7-9435-685c2e411772-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.501384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.501384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.501384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rf8j\" (UniqueName: \"kubernetes.io/projected/e9264867-1aec-45e7-9435-685c2e411772-kube-api-access-9rf8j\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.501384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.501384 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9264867-1aec-45e7-9435-685c2e411772-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.501721 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.501505    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 05 02:40:18.501721 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15397d25-488e-4ba5-8363-ea07ebe908ce-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.501721 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:40:18.501571    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls podName:15397d25-488e-4ba5-8363-ea07ebe908ce nodeName:}" failed. No retries permitted until 2026-09-05 02:40:19.001552857 +0000 UTC m=+1362.973856711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls") pod "isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" (UID: "15397d25-488e-4ba5-8363-ea07ebe908ce") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 05 02:40:18.501902 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.501885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9264867-1aec-45e7-9435-685c2e411772-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.502046 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.502024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-cabundle-cert\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.502112 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.502051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.502217 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.502200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.502266 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.502204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.503694 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.503678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9264867-1aec-45e7-9435-685c2e411772-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.509623 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.509604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdzf\" (UniqueName: \"kubernetes.io/projected/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-api-access-2fdzf\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:18.509721 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.509605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rf8j\" (UniqueName: \"kubernetes.io/projected/e9264867-1aec-45e7-9435-685c2e411772-kube-api-access-9rf8j\") pod \"isvc-canary-promote-v2-predictor-698cc558fc-r889r\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.551127 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.551075    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1397e7db-2881-4a73-89e9-c61361df9b07" path="/var/lib/kubelet/pods/1397e7db-2881-4a73-89e9-c61361df9b07/volumes"
Sep 05 02:40:18.551563 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.551550    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4cd880b-e8dc-42d6-9235-d6548c2f9c6b" path="/var/lib/kubelet/pods/e4cd880b-e8dc-42d6-9235-d6548c2f9c6b/volumes"
Sep 05 02:40:18.601853 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.601834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:18.877359 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:18.877283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"]
Sep 05 02:40:18.879577 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:40:18.879547    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9264867_1aec_45e7_9435_685c2e411772.slice/crio-bc58cc3eb5d9772822aa0be3c111f53be531b1b860f9e2a037155cfed9e945c8 WatchSource:0}: Error finding container bc58cc3eb5d9772822aa0be3c111f53be531b1b860f9e2a037155cfed9e945c8: Status 404 returned error can't find the container with id bc58cc3eb5d9772822aa0be3c111f53be531b1b860f9e2a037155cfed9e945c8
Sep 05 02:40:19.005264 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:19.005237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:19.007344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:19.007325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls\") pod \"isvc-canary-promote-predictor-6cbdffdfdb-zbk5x\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:19.031094 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:19.031059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerStarted","Data":"a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac"}
Sep 05 02:40:19.031219 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:19.031120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerStarted","Data":"bc58cc3eb5d9772822aa0be3c111f53be531b1b860f9e2a037155cfed9e945c8"}
Sep 05 02:40:19.267571 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:19.267536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:19.540102 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:19.540075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"]
Sep 05 02:40:19.542071 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:40:19.542046    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15397d25_488e_4ba5_8363_ea07ebe908ce.slice/crio-1b1aa90087085f6dd6619d2994bfef1abdde4412ac424973057371d43c902313 WatchSource:0}: Error finding container 1b1aa90087085f6dd6619d2994bfef1abdde4412ac424973057371d43c902313: Status 404 returned error can't find the container with id 1b1aa90087085f6dd6619d2994bfef1abdde4412ac424973057371d43c902313
Sep 05 02:40:20.038014 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:20.037973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerStarted","Data":"cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1"}
Sep 05 02:40:20.038414 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:20.038050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerStarted","Data":"1b1aa90087085f6dd6619d2994bfef1abdde4412ac424973057371d43c902313"}
Sep 05 02:40:23.054433 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:23.054399    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9264867-1aec-45e7-9435-685c2e411772" containerID="a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac" exitCode=0
Sep 05 02:40:23.054777 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:23.054439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerDied","Data":"a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac"}
Sep 05 02:40:24.062166 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.062127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerStarted","Data":"dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9"}
Sep 05 02:40:24.062694 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.062206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerStarted","Data":"52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8"}
Sep 05 02:40:24.062694 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.062531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:24.062694 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.062588    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:24.063808 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.063784    2579 generic.go:356] "Generic (PLEG): container finished" podID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerID="cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1" exitCode=0
Sep 05 02:40:24.063930 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.063861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerDied","Data":"cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1"}
Sep 05 02:40:24.064036 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.064012    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:40:24.083088 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:24.083047    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podStartSLOduration=2.133227665 podStartE2EDuration="6.08303301s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.949805345s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:24.081835302 +0000 UTC m=+1368.054139179" watchObservedRunningTime="2026-09-05 02:40:24.08303301 +0000 UTC m=+1368.055336885"
Sep 05 02:40:25.070900 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:25.070867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerStarted","Data":"6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606"}
Sep 05 02:40:25.071258 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:25.070927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerStarted","Data":"c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f"}
Sep 05 02:40:25.071335 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:25.071307    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:40:25.091156 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:25.091109    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podStartSLOduration=3.22043669 podStartE2EDuration="7.091096772s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.870660082s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:40:25.088718037 +0000 UTC m=+1369.061021913" watchObservedRunningTime="2026-09-05 02:40:25.091096772 +0000 UTC m=+1369.063400647"
Sep 05 02:40:30.071383 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:30.071335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:30.071988 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:30.071826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:30.073073 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:30.073038    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:40:30.077617 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:30.077595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:40:30.077801 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:30.077785    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:40:30.078196 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:30.078167    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:40:30.092621 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:30.092594    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:40:31.105956 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:31.105856    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:40:40.078176 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:40.078135    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:40:41.106074 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:41.106021    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:40:50.078821 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:50.078777    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:40:51.105398 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:40:51.105344    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:41:00.078185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:00.078143    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:41:01.105877 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:01.105824    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:41:10.078597 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:10.078550    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:41:11.105964 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:11.105914    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:41:20.078197 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:20.078149    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:41:21.105818 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:21.105769    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:41:30.078868 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:30.078766    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:41:31.106193 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:31.106145    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:41:31.545655 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:31.545617    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:41:34.545473 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:34.545419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 05 02:41:41.546879 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:41.546838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:41:44.550517 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:44.550488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:41:48.336559 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:48.336523    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"]
Sep 05 02:41:48.337001 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:48.336814    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container" containerID="cri-o://c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f" gracePeriod=30
Sep 05 02:41:48.337001 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:48.336859    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kube-rbac-proxy" containerID="cri-o://6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606" gracePeriod=30
Sep 05 02:41:48.491810 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:48.491770    2579 generic.go:356] "Generic (PLEG): container finished" podID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerID="6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606" exitCode=2
Sep 05 02:41:48.491977 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:48.491839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerDied","Data":"6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606"}
Sep 05 02:41:50.077990 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:50.077945    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 05 02:41:52.790159 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.790133    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:41:52.821733 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.821704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-rbac-proxy-sar-config\") pod \"15397d25-488e-4ba5-8363-ea07ebe908ce\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") "
Sep 05 02:41:52.821907 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.821748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15397d25-488e-4ba5-8363-ea07ebe908ce-kserve-provision-location\") pod \"15397d25-488e-4ba5-8363-ea07ebe908ce\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") "
Sep 05 02:41:52.821907 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.821785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fdzf\" (UniqueName: \"kubernetes.io/projected/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-api-access-2fdzf\") pod \"15397d25-488e-4ba5-8363-ea07ebe908ce\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") "
Sep 05 02:41:52.821907 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.821856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls\") pod \"15397d25-488e-4ba5-8363-ea07ebe908ce\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") "
Sep 05 02:41:52.822082 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.821918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-cabundle-cert\") pod \"15397d25-488e-4ba5-8363-ea07ebe908ce\" (UID: \"15397d25-488e-4ba5-8363-ea07ebe908ce\") "
Sep 05 02:41:52.822152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.822079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15397d25-488e-4ba5-8363-ea07ebe908ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15397d25-488e-4ba5-8363-ea07ebe908ce" (UID: "15397d25-488e-4ba5-8363-ea07ebe908ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:41:52.822152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.822102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15397d25-488e-4ba5-8363-ea07ebe908ce" (UID: "15397d25-488e-4ba5-8363-ea07ebe908ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:52.822257 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.822224    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:41:52.822257 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.822250    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15397d25-488e-4ba5-8363-ea07ebe908ce-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:41:52.822399 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.822378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15397d25-488e-4ba5-8363-ea07ebe908ce" (UID: "15397d25-488e-4ba5-8363-ea07ebe908ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:41:52.824009 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.823975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-api-access-2fdzf" (OuterVolumeSpecName: "kube-api-access-2fdzf") pod "15397d25-488e-4ba5-8363-ea07ebe908ce" (UID: "15397d25-488e-4ba5-8363-ea07ebe908ce"). InnerVolumeSpecName "kube-api-access-2fdzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:41:52.824159 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.824067    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15397d25-488e-4ba5-8363-ea07ebe908ce" (UID: "15397d25-488e-4ba5-8363-ea07ebe908ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:41:52.923185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.923099    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15397d25-488e-4ba5-8363-ea07ebe908ce-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:41:52.923185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.923128    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fdzf\" (UniqueName: \"kubernetes.io/projected/15397d25-488e-4ba5-8363-ea07ebe908ce-kube-api-access-2fdzf\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:41:52.923185 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:52.923145    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15397d25-488e-4ba5-8363-ea07ebe908ce-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:41:53.520103 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.520063    2579 generic.go:356] "Generic (PLEG): container finished" podID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerID="c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f" exitCode=0
Sep 05 02:41:53.520315 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.520133    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"
Sep 05 02:41:53.520315 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.520149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerDied","Data":"c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f"}
Sep 05 02:41:53.520315 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.520210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x" event={"ID":"15397d25-488e-4ba5-8363-ea07ebe908ce","Type":"ContainerDied","Data":"1b1aa90087085f6dd6619d2994bfef1abdde4412ac424973057371d43c902313"}
Sep 05 02:41:53.520315 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.520238    2579 scope.go:117] "RemoveContainer" containerID="6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606"
Sep 05 02:41:53.529271 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.529251    2579 scope.go:117] "RemoveContainer" containerID="c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f"
Sep 05 02:41:53.537288 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.537266    2579 scope.go:117] "RemoveContainer" containerID="cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1"
Sep 05 02:41:53.542637 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.542613    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"]
Sep 05 02:41:53.546512 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.546483    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6cbdffdfdb-zbk5x"]
Sep 05 02:41:53.546908 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.546892    2579 scope.go:117] "RemoveContainer" containerID="6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606"
Sep 05 02:41:53.547257 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:41:53.547236    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606\": container with ID starting with 6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606 not found: ID does not exist" containerID="6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606"
Sep 05 02:41:53.547296 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.547268    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606"} err="failed to get container status \"6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606\": rpc error: code = NotFound desc = could not find container \"6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606\": container with ID starting with 6cdba1d511c5a93a17afb41195cb1b643b7bab505a8f03affdbb41f01c2f0606 not found: ID does not exist"
Sep 05 02:41:53.547296 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.547286    2579 scope.go:117] "RemoveContainer" containerID="c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f"
Sep 05 02:41:53.547595 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:41:53.547561    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f\": container with ID starting with c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f not found: ID does not exist" containerID="c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f"
Sep 05 02:41:53.547632 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.547602    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f"} err="failed to get container status \"c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f\": rpc error: code = NotFound desc = could not find container \"c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f\": container with ID starting with c3ede3d4d51d3af823bb6452ea8e7e7a3f20baa43f96ea6a1f046ed5b5da0c0f not found: ID does not exist"
Sep 05 02:41:53.547632 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.547618    2579 scope.go:117] "RemoveContainer" containerID="cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1"
Sep 05 02:41:53.547903 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:41:53.547881    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1\": container with ID starting with cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1 not found: ID does not exist" containerID="cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1"
Sep 05 02:41:53.548003 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:53.547904    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1"} err="failed to get container status \"cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1\": rpc error: code = NotFound desc = could not find container \"cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1\": container with ID starting with cce186a351ac0560bf1735368836924b1aba06c25b32ee68c94bfdd4960363f1 not found: ID does not exist"
Sep 05 02:41:54.555763 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:41:54.555631    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" path="/var/lib/kubelet/pods/15397d25-488e-4ba5-8363-ea07ebe908ce/volumes"
Sep 05 02:43:18.591598 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:18.591568    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"]
Sep 05 02:43:18.592140 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:18.591794    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" containerID="cri-o://52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8" gracePeriod=30
Sep 05 02:43:18.592140 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:18.591830    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kube-rbac-proxy" containerID="cri-o://dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9" gracePeriod=30
Sep 05 02:43:18.936544 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:18.936479    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9264867-1aec-45e7-9435-685c2e411772" containerID="dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9" exitCode=2
Sep 05 02:43:18.936653 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:18.936552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerDied","Data":"dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9"}
Sep 05 02:43:20.071596 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:20.071557    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 05 02:43:21.546143 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:21.546100    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 05 02:43:22.437118 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.437095    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:43:22.563974 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.563950    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9264867-1aec-45e7-9435-685c2e411772-kserve-provision-location\") pod \"e9264867-1aec-45e7-9435-685c2e411772\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") "
Sep 05 02:43:22.564344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.563982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9264867-1aec-45e7-9435-685c2e411772-proxy-tls\") pod \"e9264867-1aec-45e7-9435-685c2e411772\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") "
Sep 05 02:43:22.564344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.564033    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-kube-rbac-proxy-sar-config\") pod \"e9264867-1aec-45e7-9435-685c2e411772\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") "
Sep 05 02:43:22.564344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.564050    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-cabundle-cert\") pod \"e9264867-1aec-45e7-9435-685c2e411772\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") "
Sep 05 02:43:22.564344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.564071    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rf8j\" (UniqueName: \"kubernetes.io/projected/e9264867-1aec-45e7-9435-685c2e411772-kube-api-access-9rf8j\") pod \"e9264867-1aec-45e7-9435-685c2e411772\" (UID: \"e9264867-1aec-45e7-9435-685c2e411772\") "
Sep 05 02:43:22.564344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.564241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9264867-1aec-45e7-9435-685c2e411772-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9264867-1aec-45e7-9435-685c2e411772" (UID: "e9264867-1aec-45e7-9435-685c2e411772"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 02:43:22.564344 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.564338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9264867-1aec-45e7-9435-685c2e411772" (UID: "e9264867-1aec-45e7-9435-685c2e411772"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:43:22.564585 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.564383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9264867-1aec-45e7-9435-685c2e411772" (UID: "e9264867-1aec-45e7-9435-685c2e411772"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:43:22.566082 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.566059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9264867-1aec-45e7-9435-685c2e411772-kube-api-access-9rf8j" (OuterVolumeSpecName: "kube-api-access-9rf8j") pod "e9264867-1aec-45e7-9435-685c2e411772" (UID: "e9264867-1aec-45e7-9435-685c2e411772"). InnerVolumeSpecName "kube-api-access-9rf8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:43:22.566188 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.566168    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9264867-1aec-45e7-9435-685c2e411772-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9264867-1aec-45e7-9435-685c2e411772" (UID: "e9264867-1aec-45e7-9435-685c2e411772"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:43:22.665262 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.665234    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:43:22.665262 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.665264    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9264867-1aec-45e7-9435-685c2e411772-cabundle-cert\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:43:22.665409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.665279    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rf8j\" (UniqueName: \"kubernetes.io/projected/e9264867-1aec-45e7-9435-685c2e411772-kube-api-access-9rf8j\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:43:22.665409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.665294    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9264867-1aec-45e7-9435-685c2e411772-kserve-provision-location\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:43:22.665409 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.665306    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9264867-1aec-45e7-9435-685c2e411772-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:43:22.955818 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.955744    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9264867-1aec-45e7-9435-685c2e411772" containerID="52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8" exitCode=0
Sep 05 02:43:22.955818 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.955800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"
Sep 05 02:43:22.955962 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.955820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerDied","Data":"52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8"}
Sep 05 02:43:22.955962 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.955886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r" event={"ID":"e9264867-1aec-45e7-9435-685c2e411772","Type":"ContainerDied","Data":"bc58cc3eb5d9772822aa0be3c111f53be531b1b860f9e2a037155cfed9e945c8"}
Sep 05 02:43:22.955962 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.955915    2579 scope.go:117] "RemoveContainer" containerID="dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9"
Sep 05 02:43:22.972277 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.972252    2579 scope.go:117] "RemoveContainer" containerID="52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8"
Sep 05 02:43:22.980072 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.980050    2579 scope.go:117] "RemoveContainer" containerID="a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac"
Sep 05 02:43:22.982603 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.982582    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"]
Sep 05 02:43:22.988408 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.988387    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-698cc558fc-r889r"]
Sep 05 02:43:22.988943 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.988926    2579 scope.go:117] "RemoveContainer" containerID="dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9"
Sep 05 02:43:22.989195 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:43:22.989177    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9\": container with ID starting with dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9 not found: ID does not exist" containerID="dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9"
Sep 05 02:43:22.989250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.989205    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9"} err="failed to get container status \"dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9\": rpc error: code = NotFound desc = could not find container \"dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9\": container with ID starting with dc02ffa4b2bdc45a3816c7a696c7a4c21a8ff67aa5cfe016a0e4d1874ac107b9 not found: ID does not exist"
Sep 05 02:43:22.989250 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.989221    2579 scope.go:117] "RemoveContainer" containerID="52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8"
Sep 05 02:43:22.989522 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:43:22.989502    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8\": container with ID starting with 52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8 not found: ID does not exist" containerID="52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8"
Sep 05 02:43:22.989564 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.989528    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8"} err="failed to get container status \"52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8\": rpc error: code = NotFound desc = could not find container \"52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8\": container with ID starting with 52d1da8c04e5694e6f4f68a3e7d74b1e96e5ff52286116ece475a324b6b9aad8 not found: ID does not exist"
Sep 05 02:43:22.989564 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.989545    2579 scope.go:117] "RemoveContainer" containerID="a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac"
Sep 05 02:43:22.989838 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:43:22.989820    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac\": container with ID starting with a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac not found: ID does not exist" containerID="a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac"
Sep 05 02:43:22.989885 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:22.989841    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac"} err="failed to get container status \"a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac\": rpc error: code = NotFound desc = could not find container \"a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac\": container with ID starting with a9a4308dc5ecb20aa64c94304c14fe90f8f4d0025fbc6c1ea54872b76f7873ac not found: ID does not exist"
Sep 05 02:43:24.551175 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:24.551146    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9264867-1aec-45e7-9435-685c2e411772" path="/var/lib/kubelet/pods/e9264867-1aec-45e7-9435-685c2e411772/volumes"
Sep 05 02:43:32.679135 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679106    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"]
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679437    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="storage-initializer"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679448    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="storage-initializer"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679457    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kube-rbac-proxy"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679462    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kube-rbac-proxy"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679473    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679479    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679489    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="storage-initializer"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679494    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="storage-initializer"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679503    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container"
Sep 05 02:43:32.679505 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679508    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container"
Sep 05 02:43:32.679916 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679524    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kube-rbac-proxy"
Sep 05 02:43:32.679916 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kube-rbac-proxy"
Sep 05 02:43:32.679916 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679578    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kube-rbac-proxy"
Sep 05 02:43:32.679916 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679588    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15397d25-488e-4ba5-8363-ea07ebe908ce" containerName="kserve-container"
Sep 05 02:43:32.679916 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679595    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kserve-container"
Sep 05 02:43:32.679916 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.679601    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9264867-1aec-45e7-9435-685c2e411772" containerName="kube-rbac-proxy"
Sep 05 02:43:32.682894 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.682876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.684938 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.684913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 05 02:43:32.684938 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.684932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 05 02:43:32.685223 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.685205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 05 02:43:32.685897 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.685826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-85bmm\""
Sep 05 02:43:32.686050 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.685907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 05 02:43:32.693059 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.693036    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"]
Sep 05 02:43:32.840469 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.840445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63c4314-6228-4a2c-a987-dc30b3c596cc-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.840580 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.840483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.840580 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.840541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkxf\" (UniqueName: \"kubernetes.io/projected/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-api-access-bgkxf\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.941430 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.941368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkxf\" (UniqueName: \"kubernetes.io/projected/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-api-access-bgkxf\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.941430 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.941421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63c4314-6228-4a2c-a987-dc30b3c596cc-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.941606 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.941463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.942164 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.942144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.943816 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.943796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63c4314-6228-4a2c-a987-dc30b3c596cc-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.948905 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.948884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkxf\" (UniqueName: \"kubernetes.io/projected/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-api-access-bgkxf\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hhkjh\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:32.993980 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:32.993958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:33.267165 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:33.264828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"]
Sep 05 02:43:33.267911 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:43:33.267881    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63c4314_6228_4a2c_a987_dc30b3c596cc.slice/crio-7d18adc555abff15a3a54e0acd8d2dcc6ea115bb18a3582d1827bfe1bf719448 WatchSource:0}: Error finding container 7d18adc555abff15a3a54e0acd8d2dcc6ea115bb18a3582d1827bfe1bf719448: Status 404 returned error can't find the container with id 7d18adc555abff15a3a54e0acd8d2dcc6ea115bb18a3582d1827bfe1bf719448
Sep 05 02:43:34.011398 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:34.011364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" event={"ID":"f63c4314-6228-4a2c-a987-dc30b3c596cc","Type":"ContainerStarted","Data":"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"}
Sep 05 02:43:34.011797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:34.011430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" event={"ID":"f63c4314-6228-4a2c-a987-dc30b3c596cc","Type":"ContainerStarted","Data":"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"}
Sep 05 02:43:34.011797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:34.011461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:34.011797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:34.011489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" event={"ID":"f63c4314-6228-4a2c-a987-dc30b3c596cc","Type":"ContainerStarted","Data":"7d18adc555abff15a3a54e0acd8d2dcc6ea115bb18a3582d1827bfe1bf719448"}
Sep 05 02:43:34.011797 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:34.011644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:34.013541 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:34.013516    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:43:34.030791 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:34.030750    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" podStartSLOduration=2.030733948 podStartE2EDuration="2.030733948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 02:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:43:34.029302012 +0000 UTC m=+1558.001605887" watchObservedRunningTime="2026-09-05 02:43:34.030733948 +0000 UTC m=+1558.003037826"
Sep 05 02:43:41.026752 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:43:41.026723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:45:17.815442 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:17.815411    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-hhkjh_f63c4314-6228-4a2c-a987-dc30b3c596cc/kserve-container/0.log"
Sep 05 02:45:18.053471 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.053430    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"]
Sep 05 02:45:18.053758 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.053732    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kserve-container" containerID="cri-o://277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57" gracePeriod=30
Sep 05 02:45:18.053888 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.053774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kube-rbac-proxy" containerID="cri-o://f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6" gracePeriod=30
Sep 05 02:45:18.298145 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.298123    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:45:18.346907 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.346844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-rbac-proxy-sar-config\") pod \"f63c4314-6228-4a2c-a987-dc30b3c596cc\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") "
Sep 05 02:45:18.346907 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.346875    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgkxf\" (UniqueName: \"kubernetes.io/projected/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-api-access-bgkxf\") pod \"f63c4314-6228-4a2c-a987-dc30b3c596cc\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") "
Sep 05 02:45:18.347093 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.346934    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63c4314-6228-4a2c-a987-dc30b3c596cc-proxy-tls\") pod \"f63c4314-6228-4a2c-a987-dc30b3c596cc\" (UID: \"f63c4314-6228-4a2c-a987-dc30b3c596cc\") "
Sep 05 02:45:18.347237 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.347213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f63c4314-6228-4a2c-a987-dc30b3c596cc" (UID: "f63c4314-6228-4a2c-a987-dc30b3c596cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 02:45:18.348931 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.348905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63c4314-6228-4a2c-a987-dc30b3c596cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f63c4314-6228-4a2c-a987-dc30b3c596cc" (UID: "f63c4314-6228-4a2c-a987-dc30b3c596cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 02:45:18.348931 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.348926    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-api-access-bgkxf" (OuterVolumeSpecName: "kube-api-access-bgkxf") pod "f63c4314-6228-4a2c-a987-dc30b3c596cc" (UID: "f63c4314-6228-4a2c-a987-dc30b3c596cc"). InnerVolumeSpecName "kube-api-access-bgkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 02:45:18.448134 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.448109    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f63c4314-6228-4a2c-a987-dc30b3c596cc-proxy-tls\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:45:18.448134 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.448129    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:45:18.448254 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.448138    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkxf\" (UniqueName: \"kubernetes.io/projected/f63c4314-6228-4a2c-a987-dc30b3c596cc-kube-api-access-bgkxf\") on node \"ip-10-0-138-91.ec2.internal\" DevicePath \"\""
Sep 05 02:45:18.519795 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.519769    2579 generic.go:356] "Generic (PLEG): container finished" podID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerID="f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6" exitCode=2
Sep 05 02:45:18.519795 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.519794    2579 generic.go:356] "Generic (PLEG): container finished" podID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerID="277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57" exitCode=2
Sep 05 02:45:18.519935 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.519824    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"
Sep 05 02:45:18.519935 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.519830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" event={"ID":"f63c4314-6228-4a2c-a987-dc30b3c596cc","Type":"ContainerDied","Data":"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"}
Sep 05 02:45:18.519935 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.519888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" event={"ID":"f63c4314-6228-4a2c-a987-dc30b3c596cc","Type":"ContainerDied","Data":"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"}
Sep 05 02:45:18.519935 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.519920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh" event={"ID":"f63c4314-6228-4a2c-a987-dc30b3c596cc","Type":"ContainerDied","Data":"7d18adc555abff15a3a54e0acd8d2dcc6ea115bb18a3582d1827bfe1bf719448"}
Sep 05 02:45:18.520085 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.519956    2579 scope.go:117] "RemoveContainer" containerID="f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"
Sep 05 02:45:18.528100 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.528083    2579 scope.go:117] "RemoveContainer" containerID="277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"
Sep 05 02:45:18.535135 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.535117    2579 scope.go:117] "RemoveContainer" containerID="f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"
Sep 05 02:45:18.535385 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:45:18.535368    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6\": container with ID starting with f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6 not found: ID does not exist" containerID="f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"
Sep 05 02:45:18.535433 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.535392    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"} err="failed to get container status \"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6\": rpc error: code = NotFound desc = could not find container \"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6\": container with ID starting with f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6 not found: ID does not exist"
Sep 05 02:45:18.535433 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.535408    2579 scope.go:117] "RemoveContainer" containerID="277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"
Sep 05 02:45:18.535652 ip-10-0-138-91 kubenswrapper[2579]: E0905 02:45:18.535631    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57\": container with ID starting with 277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57 not found: ID does not exist" containerID="277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"
Sep 05 02:45:18.535730 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.535655    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"} err="failed to get container status \"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57\": rpc error: code = NotFound desc = could not find container \"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57\": container with ID starting with 277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57 not found: ID does not exist"
Sep 05 02:45:18.535730 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.535684    2579 scope.go:117] "RemoveContainer" containerID="f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"
Sep 05 02:45:18.535884 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.535868    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6"} err="failed to get container status \"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6\": rpc error: code = NotFound desc = could not find container \"f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6\": container with ID starting with f42b7c2578a1a9b60ec86ce2e75b691cd5fe7d278ed2ca59294760562b0ff8b6 not found: ID does not exist"
Sep 05 02:45:18.535929 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.535884    2579 scope.go:117] "RemoveContainer" containerID="277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"
Sep 05 02:45:18.536105 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.536089    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57"} err="failed to get container status \"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57\": rpc error: code = NotFound desc = could not find container \"277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57\": container with ID starting with 277511e9f482dc0eb013af4a8de70b7b65d482120afbd30dc77a238f3e90fd57 not found: ID does not exist"
Sep 05 02:45:18.542649 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.542625    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"]
Sep 05 02:45:18.550099 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:18.550072    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hhkjh"]
Sep 05 02:45:20.550521 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:20.550487    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" path="/var/lib/kubelet/pods/f63c4314-6228-4a2c-a987-dc30b3c596cc/volumes"
Sep 05 02:45:53.092179 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:53.092151    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c75c8_4183dcc7-2263-45e0-97e8-ad25a6c7ad9a/global-pull-secret-syncer/0.log"
Sep 05 02:45:53.277118 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:53.277077    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tqks4_00d08ef6-5502-4c36-9da3-e33abed6ba56/konnectivity-agent/0.log"
Sep 05 02:45:53.345880 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:53.345805    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-91.ec2.internal_8af2bf76f7435a43939d2bafa383cafd/haproxy/0.log"
Sep 05 02:45:57.463068 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.462976    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-tqkvz_57e56495-a3b3-4d73-b40d-001609204804/cluster-monitoring-operator/0.log"
Sep 05 02:45:57.494179 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.494094    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-242ct_30d67f0e-58fc-46f2-8e09-d68e9a20d747/kube-state-metrics/0.log"
Sep 05 02:45:57.519308 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.519230    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-242ct_30d67f0e-58fc-46f2-8e09-d68e9a20d747/kube-rbac-proxy-main/0.log"
Sep 05 02:45:57.547060 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.546992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-242ct_30d67f0e-58fc-46f2-8e09-d68e9a20d747/kube-rbac-proxy-self/0.log"
Sep 05 02:45:57.607444 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.607376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-lztgm_9d27d2fa-7c74-4d65-8554-56b9909d589c/monitoring-plugin/0.log"
Sep 05 02:45:57.871426 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.871348    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x57pm_5d817548-f439-4d19-b3c5-c74c8a1c92f8/node-exporter/0.log"
Sep 05 02:45:57.892015 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.891954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x57pm_5d817548-f439-4d19-b3c5-c74c8a1c92f8/kube-rbac-proxy/0.log"
Sep 05 02:45:57.912260 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.912195    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x57pm_5d817548-f439-4d19-b3c5-c74c8a1c92f8/init-textfile/0.log"
Sep 05 02:45:57.944657 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.944579    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-w6f6b_d9a4adad-8278-4253-a58e-a20c3f4464f6/kube-rbac-proxy-main/0.log"
Sep 05 02:45:57.969343 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.969274    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-w6f6b_d9a4adad-8278-4253-a58e-a20c3f4464f6/kube-rbac-proxy-self/0.log"
Sep 05 02:45:57.992970 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:57.992910    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-w6f6b_d9a4adad-8278-4253-a58e-a20c3f4464f6/openshift-state-metrics/0.log"
Sep 05 02:45:58.325277 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:58.325210    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-wk789_1bb1005d-8930-4403-bee7-7bcb5475deab/prometheus-operator-admission-webhook/0.log"
Sep 05 02:45:58.365255 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:58.365193    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dd57d64b7-g76gj_0c549329-240e-49d6-965c-aa1d755f5311/telemeter-client/0.log"
Sep 05 02:45:58.405566 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:58.405499    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dd57d64b7-g76gj_0c549329-240e-49d6-965c-aa1d755f5311/reload/0.log"
Sep 05 02:45:58.436009 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:45:58.435949    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dd57d64b7-g76gj_0c549329-240e-49d6-965c-aa1d755f5311/kube-rbac-proxy/0.log"
Sep 05 02:46:00.872871 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:00.872842    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c9b6d7b79-72kh2_7916ab3d-d846-4260-a6f6-56ffcbd18771/console/0.log"
Sep 05 02:46:00.919870 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:00.919844    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-8wk42_4bf3cac2-d91a-49b2-9922-8484815eaf7d/download-server/0.log"
Sep 05 02:46:01.632606 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.632574    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"]
Sep 05 02:46:01.632937 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.632923    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kserve-container"
Sep 05 02:46:01.632994 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.632939    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kserve-container"
Sep 05 02:46:01.632994 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.632946    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kube-rbac-proxy"
Sep 05 02:46:01.632994 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.632953    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kube-rbac-proxy"
Sep 05 02:46:01.633092 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.633013    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kserve-container"
Sep 05 02:46:01.633092 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.633021    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63c4314-6228-4a2c-a987-dc30b3c596cc" containerName="kube-rbac-proxy"
Sep 05 02:46:01.636134 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.636114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.638498 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.638467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s672k\"/\"openshift-service-ca.crt\""
Sep 05 02:46:01.638498 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.638492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s672k\"/\"default-dockercfg-d5z68\""
Sep 05 02:46:01.639342 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.639327    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s672k\"/\"kube-root-ca.crt\""
Sep 05 02:46:01.646064 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.646042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"]
Sep 05 02:46:01.802098 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.802076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-proc\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.802216 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.802109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-lib-modules\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.802216 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.802131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxnk\" (UniqueName: \"kubernetes.io/projected/87ea963f-fe41-46f1-9892-ec5071cd3629-kube-api-access-hmxnk\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.802216 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.802148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-sys\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.802327 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.802223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-podres\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903230 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-proc\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903230 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-lib-modules\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxnk\" (UniqueName: \"kubernetes.io/projected/87ea963f-fe41-46f1-9892-ec5071cd3629-kube-api-access-hmxnk\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-sys\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-proc\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-podres\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-lib-modules\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-sys\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.903643 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.903480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87ea963f-fe41-46f1-9892-ec5071cd3629-podres\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.911306 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.911283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxnk\" (UniqueName: \"kubernetes.io/projected/87ea963f-fe41-46f1-9892-ec5071cd3629-kube-api-access-hmxnk\") pod \"perf-node-gather-daemonset-7srs6\" (UID: \"87ea963f-fe41-46f1-9892-ec5071cd3629\") " pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:01.946743 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:01.946720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:02.219787 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.219764    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"]
Sep 05 02:46:02.222125 ip-10-0-138-91 kubenswrapper[2579]: W0905 02:46:02.222099    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87ea963f_fe41_46f1_9892_ec5071cd3629.slice/crio-5969f9941fce7c738d76851bc79ff17e9fc07d68debca587c03f450f0d4d430d WatchSource:0}: Error finding container 5969f9941fce7c738d76851bc79ff17e9fc07d68debca587c03f450f0d4d430d: Status 404 returned error can't find the container with id 5969f9941fce7c738d76851bc79ff17e9fc07d68debca587c03f450f0d4d430d
Sep 05 02:46:02.383139 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.383115    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6b8vv_7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e/dns/0.log"
Sep 05 02:46:02.406950 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.406914    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6b8vv_7e3623ea-47ad-4dd5-b6c6-9f892eb11c2e/kube-rbac-proxy/0.log"
Sep 05 02:46:02.519528 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.519503    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6sbhk_a7eed417-1f97-47b2-b2e2-6ec960bbfda3/dns-node-resolver/0.log"
Sep 05 02:46:02.745355 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.745327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6" event={"ID":"87ea963f-fe41-46f1-9892-ec5071cd3629","Type":"ContainerStarted","Data":"fcb851345cf6cb3fc48316a8f5491f5f729db8427aea949031302e30c77ab6ea"}
Sep 05 02:46:02.745510 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.745372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6" event={"ID":"87ea963f-fe41-46f1-9892-ec5071cd3629","Type":"ContainerStarted","Data":"5969f9941fce7c738d76851bc79ff17e9fc07d68debca587c03f450f0d4d430d"}
Sep 05 02:46:02.745510 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.745403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:02.763333 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:02.763285    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6" podStartSLOduration=1.763270946 podStartE2EDuration="1.763270946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 02:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 02:46:02.761677268 +0000 UTC m=+1706.733981163" watchObservedRunningTime="2026-09-05 02:46:02.763270946 +0000 UTC m=+1706.735574821"
Sep 05 02:46:03.049739 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:03.049711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2mzt5_d8f03596-1d37-470e-86e3-0b2ae5138bdf/node-ca/0.log"
Sep 05 02:46:04.204617 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:04.204589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62k4n_deeaa7ba-c6e9-461a-8163-c1d4d296f225/serve-healthcheck-canary/0.log"
Sep 05 02:46:04.584130 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:04.584095    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-x7g2f_57df1726-1a5b-4ca3-828f-3c39ccbb533c/insights-operator/0.log"
Sep 05 02:46:04.584515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:04.584495    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-x7g2f_57df1726-1a5b-4ca3-828f-3c39ccbb533c/insights-operator/1.log"
Sep 05 02:46:04.670515 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:04.670492    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glz8c_a1b9c92b-50cc-4a19-bc24-5643ce7aec0f/kube-rbac-proxy/0.log"
Sep 05 02:46:04.689729 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:04.689705    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glz8c_a1b9c92b-50cc-4a19-bc24-5643ce7aec0f/exporter/0.log"
Sep 05 02:46:04.708652 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:04.708633    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-glz8c_a1b9c92b-50cc-4a19-bc24-5643ce7aec0f/extractor/0.log"
Sep 05 02:46:07.072461 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:07.072434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-djbsn_ac2fa516-9ee4-4e90-a6cc-eff78d8e250d/server/0.log"
Sep 05 02:46:07.184549 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:07.184521    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-jj6d9_4dec06be-89c7-4722-bdee-0b0a2764de45/manager/0.log"
Sep 05 02:46:07.258406 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:07.258362    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-r2hv6_574c5e5a-a96e-42d7-b1f5-eb9df2e315fb/seaweedfs/0.log"
Sep 05 02:46:08.760736 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:08.760709    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s672k/perf-node-gather-daemonset-7srs6"
Sep 05 02:46:13.159400 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.159376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jm6h_cb5c424f-d9a6-4d3c-a4b1-c26857227b44/kube-multus-additional-cni-plugins/0.log"
Sep 05 02:46:13.188355 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.188334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jm6h_cb5c424f-d9a6-4d3c-a4b1-c26857227b44/egress-router-binary-copy/0.log"
Sep 05 02:46:13.225268 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.225248    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jm6h_cb5c424f-d9a6-4d3c-a4b1-c26857227b44/cni-plugins/0.log"
Sep 05 02:46:13.254850 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.254833    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jm6h_cb5c424f-d9a6-4d3c-a4b1-c26857227b44/bond-cni-plugin/0.log"
Sep 05 02:46:13.284479 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.284460    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jm6h_cb5c424f-d9a6-4d3c-a4b1-c26857227b44/routeoverride-cni/0.log"
Sep 05 02:46:13.307994 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.307972    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jm6h_cb5c424f-d9a6-4d3c-a4b1-c26857227b44/whereabouts-cni-bincopy/0.log"
Sep 05 02:46:13.334353 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.334324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7jm6h_cb5c424f-d9a6-4d3c-a4b1-c26857227b44/whereabouts-cni/0.log"
Sep 05 02:46:13.746290 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.746267    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hk2dc_93b93690-899f-4fcc-a6da-cb1fe002b30b/kube-multus/0.log"
Sep 05 02:46:13.879266 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.879245    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xk94l_4662252d-cd26-4d5c-ad62-0c815a71e462/network-metrics-daemon/0.log"
Sep 05 02:46:13.897144 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:13.897123    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xk94l_4662252d-cd26-4d5c-ad62-0c815a71e462/kube-rbac-proxy/0.log"
Sep 05 02:46:14.853532 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:14.853504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/ovn-controller/0.log"
Sep 05 02:46:14.872810 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:14.872781    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/ovn-acl-logging/0.log"
Sep 05 02:46:14.896686 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:14.896652    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/kube-rbac-proxy-node/0.log"
Sep 05 02:46:14.921487 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:14.921464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/kube-rbac-proxy-ovn-metrics/0.log"
Sep 05 02:46:14.939536 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:14.939517    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/northd/0.log"
Sep 05 02:46:14.960152 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:14.960131    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/nbdb/0.log"
Sep 05 02:46:14.983301 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:14.983281    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/sbdb/0.log"
Sep 05 02:46:15.024269 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:15.024250    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnk6d_a4d66a85-c9db-43a2-8465-62b8a630df97/ovnkube-controller/0.log"
Sep 05 02:46:16.208079 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:16.208048    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-8tdx7_79530a70-c1d2-4bf0-ad1c-a3c3af0cce86/check-endpoints/0.log"
Sep 05 02:46:16.229385 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:16.229361    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-svblx_59e79702-89d7-49a0-bd00-8754b43eb10d/network-check-target-container/0.log"
Sep 05 02:46:17.231595 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:17.231566    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-s4qjf_88034fda-9465-4196-bd5e-9ca3907b9bbb/iptables-alerter/0.log"
Sep 05 02:46:17.906228 ip-10-0-138-91 kubenswrapper[2579]: I0905 02:46:17.906153    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-6md2v_5150c3b3-98c7-4bd7-b469-950ca5c40029/tuned/0.log"