Sep 09 20:50:11.300301 ip-10-0-129-205 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:50:11.300311 ip-10-0-129-205 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:50:11.300321 ip-10-0-129-205 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:50:11.301001 ip-10-0-129-205 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:50:21.394964 ip-10-0-129-205 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:50:21.394978 ip-10-0-129-205 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c5209a9b068f430e98ba1d6e154d793a --
Sep 09 20:52:07.659990 ip-10-0-129-205 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:52:08.113833 ip-10-0-129-205 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:08.113833 ip-10-0-129-205 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:52:08.113833 ip-10-0-129-205 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:08.113833 ip-10-0-129-205 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:52:08.113833 ip-10-0-129-205 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:52:08.114905 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.114816    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:52:08.119186 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119165    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:08.119186 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119183    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:08.119186 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119187    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:08.119186 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119190    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:08.119186 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119193    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119196    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119200    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119202    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119208    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119212    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119216    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119220    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119224    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119227    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119230    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119233    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119236    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119240    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119244    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119246    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119249    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119252    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119255    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:08.119377 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119258    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119260    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119263    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119266    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119269    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119272    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119276    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119279    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119282    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119284    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119287    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119290    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119293    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119295    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119298    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119301    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119304    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119306    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119309    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119311    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:08.119882 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119314    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119316    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119319    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119321    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119324    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119327    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119329    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119332    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119334    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119337    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119339    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119342    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119344    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119347    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119349    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119352    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119354    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119357    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119359    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119362    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:08.120368 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119364    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119367    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119369    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119372    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119374    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119378    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119381    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119384    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119387    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119390    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119392    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119395    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119397    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119400    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119402    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119405    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119407    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119410    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119412    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119415    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:08.120876 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119418    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119421    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119423    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119426    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119428    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119430    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119433    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119436    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119438    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119441    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119443    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119446    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119449    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119451    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119454    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.119456    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120162    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120168    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120171    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:08.121350 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120175    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120178    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120181    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120183    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120186    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120189    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120191    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120194    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120197    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120199    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120201    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120204    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120207    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120209    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120212    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120215    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120217    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120220    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120223    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120225    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:08.121820 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120228    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120230    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120233    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120237    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120242    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120246    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120249    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120252    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120255    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120257    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120260    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120263    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120266    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120269    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120271    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120274    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120277    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120279    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120281    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:08.122340 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120284    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120287    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120289    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120292    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120294    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120297    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120299    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120302    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120305    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120307    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120310    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120312    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120314    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120317    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120319    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120322    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120325    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120328    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120330    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120333    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:08.122811 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120336    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120339    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120342    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120345    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120347    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120350    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120352    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120355    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120358    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120361    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120363    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120366    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120368    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120371    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120373    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120376    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120378    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120381    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120383    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:08.123304 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120386    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120388    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120392    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120395    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120397    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120400    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120402    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120405    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120408    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120411    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120413    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120416    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120418    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120421    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120423    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120426    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120428    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120431    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120842    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120847    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:08.123783 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120850    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120853    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120856    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120858    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120861    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120863    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120866    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120868    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120871    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120874    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120876    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120879    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120882    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120885    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120888    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120891    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120893    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120896    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120899    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120903    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:08.124261 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120906    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120909    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120911    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120914    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120916    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120918    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120921    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120923    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120925    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120928    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120930    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120933    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120935    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120937    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120940    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120942    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120944    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120947    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120950    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120952    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:08.124755 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120955    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120957    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120967    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120969    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120972    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120974    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120977    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120979    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120982    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120985    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120988    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120990    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120992    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120995    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.120998    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121000    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121002    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121005    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121007    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121010    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:08.125229 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121012    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121014    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121017    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121019    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121021    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121024    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121028    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121031    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121034    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121037    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121039    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121042    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121044    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121047    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121050    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121053    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121055    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121058    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121060    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:08.125731 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121063    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121066    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121068    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121071    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121073    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121076    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121078    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121081    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121083    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121085    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121088    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121092    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121095    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121097    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121100    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121102    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121105    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121107    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121178    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121210    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:52:08.126187 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121216    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121221    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121224    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121228    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121232    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121236    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121239    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121242    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121245    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121252    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121255    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121258    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121260    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121263    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121266    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121269    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121273    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121276    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121279    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121282    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121285    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121288    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121291    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:52:08.126683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121294    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121297    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121300    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121303    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121306    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121309    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121312    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121316    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121318    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121321    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121324    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121326    2573 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121329    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121333    2573 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121337    2573 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121339    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121342    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121345    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121352    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121355    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121357    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121360    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121363    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121366    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121369    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:52:08.127219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121372    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121375    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121377    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121380    2573 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121383    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121386    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121389    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121392    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121395    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121398    2573 flags.go:64] FLAG: --help="false"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121401    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121404    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121407    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121410    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121412    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121416    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121418    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121421    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121424    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121426    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121429    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121432    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121434    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121437    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:52:08.127908 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121440    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121445    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121448    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121451    2573 flags.go:64] FLAG: --lock-file=""
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121453    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121456    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121459    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121464    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121468    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121470    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121473    2573 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121476    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121479    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121482    2573 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121484    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121488    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121491    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121495    2573 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121497    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121500    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121503    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121506    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121508    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121511    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121514    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:52:08.128496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121521    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121525    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121527    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121530    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121533    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121538    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121540    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121543    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121547    2573 flags.go:64] FLAG: --port="10250"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121565    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121568    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-054263a3a482c8fea"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121571    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121574    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121577    2573 flags.go:64] FLAG: --register-node="true"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121579    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121583    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121587    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121589    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121592    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121596    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121599    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121602    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121604    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121607    2573 flags.go:64] FLAG: --runonce="false"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121610    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:52:08.129232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121613    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121616    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121618    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121621    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121624    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121627    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121630    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121632    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121635    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121638    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121640    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121643    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121646    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121649    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121653    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121657    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121661    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121665    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121668    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121670    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121673    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121676    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121679    2573 flags.go:64] FLAG: --v="2"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121683    2573 flags.go:64] FLAG: --version="false"
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121687    2573 flags.go:64] FLAG: --vmodule=""
Sep 09 20:52:08.129861 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121691    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.121694    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121803    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121807    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121810    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121812    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121815    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121818    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121820    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121823    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121826    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121828    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121831    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121834    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121836    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121839    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121841    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121843    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121846    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121848    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:08.130483 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121851    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121853    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121856    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121860    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121862    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121864    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121867    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121870    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121872    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121875    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121878    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121880    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121883    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121886    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121888    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121890    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121893    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121895    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121898    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121900    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:08.130980 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121903    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121905    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121907    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121910    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121912    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121915    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121917    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121920    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121922    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121924    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121927    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121929    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121933    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121936    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121941    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121945    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121947    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121950    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121952    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121955    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:08.131470 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121958    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121961    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121963    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121965    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121968    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121971    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121973    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121976    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121978    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121981    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121983    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121986    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121988    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121990    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121993    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121995    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.121999    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122002    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122005    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122007    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:08.132040 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122009    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122012    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122014    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122016    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122019    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122021    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122025    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122030    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122033    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122036    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122038    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122041    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122044    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122047    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122050    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122052    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122055    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122058    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122061    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:08.132523 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122063    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.122066    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.122843    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.130349    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.130475    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.131546    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.131570    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131631    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131636    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131640    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131643    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131647    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131651    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:08.133007 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131654    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131658    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131661    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131664    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131668    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131671    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131674    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131676    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131679    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131682    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131685    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131688    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131690    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131693    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131695    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131698    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131700    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131703    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131706    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:08.133338 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131709    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131711    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131714    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131716    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131720    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131722    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131725    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131728    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131730    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131733    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131735    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131738    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131740    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131743    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131745    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131748    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131750    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131753    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131755    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131757    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:08.133810 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131760    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131763    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131765    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131767    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131771    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131775    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131777    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131780    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131782    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131785    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131787    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131790    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131793    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131795    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131798    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131800    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131804    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131807    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131809    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:08.134284 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131812    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131814    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131817    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131819    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131822    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131824    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131826    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131829    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131831    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131834    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131837    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131839    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131842    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131844    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131846    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131849    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131851    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131854    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131856    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:08.134777 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131859    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131861    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131864    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131866    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131869    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131871    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131874    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131877    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131879    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131881    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131884    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131887    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131890    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131892    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131895    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:08.135274 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.131897    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.131902    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132009    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132013    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132016    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132018    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132021    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132024    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132026    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132029    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132031    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132034    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132037    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132039    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:52:08.135700 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132043    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132047    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132050    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132053    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132056    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132058    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132062    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132065    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132067    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132070    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132072    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132076    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132078    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132081    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132084    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132087    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132089    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132091    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132094    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132096    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:52:08.136156 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132099    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132101    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132104    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132106    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132109    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132111    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132115    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132118    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132120    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132123    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132126    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132129    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132131    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132134    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132136    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132139    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132141    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132144    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132146    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:52:08.136697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132149    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132151    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132153    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132156    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132158    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132161    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132165    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132167    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132170    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132172    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132175    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132177    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132179    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132182    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132184    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132187    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132190    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132193    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132196    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132198    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:52:08.137194 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132201    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132203    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132206    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132208    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132211    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132213    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132216    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132218    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132220    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132223    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132225    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132228    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132230    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132233    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132235    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132238    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132240    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132242    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132245    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132248    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:52:08.137697 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132251    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132253    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132256    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132258    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132261    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132263    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132266    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:08.132268    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.132273    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.132372    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.134705    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.135682    2573 server.go:1019] "Starting client certificate rotation"
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.135800    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:08.138282 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.135850    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:52:08.138607 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.138594    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:08.145759 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.145736    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:52:08.158930 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.158911    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:52:08.162224 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.162210    2573 log.go:25] "Validated CRI v1 image API"
Sep 09 20:52:08.163084 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.163068    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:52:08.168848 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.168825    2573 fs.go:135] Filesystem UUIDs: map[6ad58e04-5dc2-4b31-98ae-43861866641b:/dev/nvme0n1p4 76cdd3ee-87c0-4ce4-a5bd-9728942e7d86:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 20:52:08.168936 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.168846    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:52:08.169640 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.169618    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:08.173161 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.173040    2573 manager.go:216] Machine: {Timestamp:2026-09-09 20:52:08.172544381 +0000 UTC m=+0.383088157 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3179300 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f746dd4824a51228b95f2e0dfbc89 SystemUUID:ec2f746d-d482-4a51-228b-95f2e0dfbc89 BootID:c5209a9b-068f-430e-98ba-1d6e154d793a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bd:4a:fc:2b:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bd:4a:fc:2b:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:c4:60:57:c6:0f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:52:08.173161 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.173152    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:52:08.173328 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.173311    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:52:08.177260 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.177238    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:52:08.177400 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.177263    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:52:08.178062 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.178052    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:52:08.178101 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.178069    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:52:08.178101 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.178077    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:08.178101 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.178090    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:52:08.179809 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.179799    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:52:08.184826 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.184814    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:08.184938 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.184929    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:52:08.184970 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.184960    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:52:08.184999 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.184970    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:52:08.184999 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.184985    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:52:08.184999 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.184994    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:52:08.185116 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.185009    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:52:08.185446 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.185432    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:08.185481 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.185459    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:52:08.190891 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.190864    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:52:08.194051 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194035    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:52:08.194118 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194065    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:52:08.194225 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194215    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:52:08.194281 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194229    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:52:08.194281 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194235    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:52:08.194281 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194240    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:52:08.194281 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194248    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:52:08.194281 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194254    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:52:08.194937 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194928    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:52:08.194937 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194937    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:52:08.195040 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194944    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:52:08.195040 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194950    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:52:08.195040 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194958    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:52:08.195040 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.194973    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:52:08.195684 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.195665    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:52:08.195760 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.195699    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:52:08.195974 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.195962    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:52:08.196007 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.195975    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:52:08.197566 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.197539    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-205.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:52:08.199901 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.199889    2573 server.go:1268] "Started kubelet"
Sep 09 20:52:08.199991 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.199958    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:52:08.200090 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.199993    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:52:08.200090 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.200064    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:52:08.200796 ip-10-0-129-205 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:52:08.201515 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.201494    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:52:08.202158 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.202146    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:52:08.207155 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.207137    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:08.207837 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.207818    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:52:08.208156 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.208134    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.208235 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.208173    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:52:08.208612 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.208595    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:52:08.208800 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.208787    2573 factory.go:55] Registering systemd factory
Sep 09 20:52:08.208859 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.208804    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:52:08.208913 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.208861    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:52:08.208913 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.208875    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:52:08.209066 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.209051    2573 factory.go:153] Registering CRI-O factory
Sep 09 20:52:08.209131 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.209070    2573 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:52:08.209179 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.209139    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:52:08.209179 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.209158    2573 factory.go:103] Registering Raw factory
Sep 09 20:52:08.209263 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.209251    2573 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:52:08.209923 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.209910    2573 manager.go:349] Starting recovery of all containers
Sep 09 20:52:08.210610 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.210589    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:52:08.210610 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.210606    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:52:08.218034 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.217997    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:52:08.218167 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.218135    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:52:08.219764 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.218249    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-205.ec2.internal.18d3c2508c195914  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-205.ec2.internal,UID:ip-10-0-129-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-205.ec2.internal,},FirstTimestamp:2026-09-09 20:52:08.199862548 +0000 UTC m=+0.410406323,LastTimestamp:2026-09-09 20:52:08.199862548 +0000 UTC m=+0.410406323,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-205.ec2.internal,}"
Sep 09 20:52:08.222974 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.222956    2573 manager.go:354] Recovery completed
Sep 09 20:52:08.224464 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.224446    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 20:52:08.227570 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.227540    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:08.232438 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.232410    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:08.232587 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.232448    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:08.232587 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.232458    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:08.233167 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.233153    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:52:08.233167 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.233164    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:08.233255 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.233166    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:52:08.233255 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.233231    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:52:08.234399 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.234385    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9gpff"
Sep 09 20:52:08.234820 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.234754    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-205.ec2.internal.18d3c2508e0a623f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-205.ec2.internal,UID:ip-10-0-129-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-205.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-205.ec2.internal,},FirstTimestamp:2026-09-09 20:52:08.232436287 +0000 UTC m=+0.442980065,LastTimestamp:2026-09-09 20:52:08.232436287 +0000 UTC m=+0.442980065,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-205.ec2.internal,}"
Sep 09 20:52:08.235381 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.235361    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:08.235448 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.235389    2573 policy_none.go:49] "None policy: Start"
Sep 09 20:52:08.235448 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.235393    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:08.235448 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.235401    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:52:08.235448 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.235404    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:08.235448 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.235411    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:52:08.237512 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.237501    2573 policy_none.go:47] "Start"
Sep 09 20:52:08.241855 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.241840    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9gpff"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.277169    2573 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.277210    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.277220    2573 server.go:87] "Starting device plugin registration server"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.277489    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.277519    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.277604    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.277685    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.277694    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.278300    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.278366    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.280048    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.281683    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.281764    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.281796    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.281830    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:52:08.291320 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.283944    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:08.378352 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.378255    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:08.379260 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.379246    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:08.379318 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.379277    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:08.379318 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.379289    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:08.379378 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.379320    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.382603 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.382576    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"]
Sep 09 20:52:08.382660 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.382654    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:08.386857 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.386834    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:08.386857 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.386858    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:08.387002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.386868    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:08.389129 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.389117    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:08.389248 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.389233    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.389303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.389265    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:08.389371 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.389360    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.389407 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.389378    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-205.ec2.internal\": node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.390049 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.390029    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:08.390049 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.390043    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:08.390179 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.390059    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:08.390179 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.390066    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:08.390179 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.390072    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:08.390179 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.390075    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:08.392354 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.392339    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.392436 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.392362    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:52:08.393040 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.393020    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:52:08.393040 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.393050    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:52:08.393163 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.393065    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:52:08.411934 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.411917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c5e0ae4901bfc17af41e47ecf4fc10e2-config\") pod \"kube-apiserver-proxy-ip-10-0-129-205.ec2.internal\" (UID: \"c5e0ae4901bfc17af41e47ecf4fc10e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.412030 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.411940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c50a8950375d1e1a79c4ab902dfc9e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal\" (UID: \"1c50a8950375d1e1a79c4ab902dfc9e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.412030 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.411956    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c50a8950375d1e1a79c4ab902dfc9e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal\" (UID: \"1c50a8950375d1e1a79c4ab902dfc9e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.422226 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.422203    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-205.ec2.internal\" not found" node="ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.426691 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.426675    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-205.ec2.internal\" not found" node="ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.428437 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.428421    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.512789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.512756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c50a8950375d1e1a79c4ab902dfc9e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal\" (UID: \"1c50a8950375d1e1a79c4ab902dfc9e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.512789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.512802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c50a8950375d1e1a79c4ab902dfc9e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal\" (UID: \"1c50a8950375d1e1a79c4ab902dfc9e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.513024 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.512827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c5e0ae4901bfc17af41e47ecf4fc10e2-config\") pod \"kube-apiserver-proxy-ip-10-0-129-205.ec2.internal\" (UID: \"c5e0ae4901bfc17af41e47ecf4fc10e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.513024 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.512867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c50a8950375d1e1a79c4ab902dfc9e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal\" (UID: \"1c50a8950375d1e1a79c4ab902dfc9e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.513024 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.512875    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c5e0ae4901bfc17af41e47ecf4fc10e2-config\") pod \"kube-apiserver-proxy-ip-10-0-129-205.ec2.internal\" (UID: \"c5e0ae4901bfc17af41e47ecf4fc10e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.513024 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.512865    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c50a8950375d1e1a79c4ab902dfc9e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal\" (UID: \"1c50a8950375d1e1a79c4ab902dfc9e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.528863 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.528842    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.629419 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.629342    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.726166 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.726128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.729816 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.729793    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.730366 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:08.730351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:08.829916 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.829876    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:08.930466 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:08.930381    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:09.031004 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.030971    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:09.131505 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.131460    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-205.ec2.internal\" not found"
Sep 09 20:52:09.135613 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.135598    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:52:09.135781 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.135764    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:52:09.181863 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.181809    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:09.185404 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.185378    2573 apiserver.go:52] "Watching apiserver"
Sep 09 20:52:09.199885 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.199855    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:52:09.200265 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.200239    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-k2kqd","kube-system/konnectivity-agent-l8c65","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl","openshift-image-registry/node-ca-xtcgf","openshift-multus/multus-additional-cni-plugins-q5fd2","openshift-multus/multus-g7mfh","openshift-multus/network-metrics-daemon-xr4nz","openshift-network-operator/iptables-alerter-vn5xq","openshift-ovn-kubernetes/ovnkube-node-lbk97","kube-system/global-pull-secret-syncer-c45vx","openshift-cluster-node-tuning-operator/tuned-fbj89","openshift-dns/node-resolver-dq49d"]
Sep 09 20:52:09.202808 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.202788    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:09.202911 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.202864    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:09.204863 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.204844    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.207019 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.206999    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.207111 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.207003    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.207111 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.207049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:09.207209 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.207106    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:09.207209 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.207006    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8n2qj\""
Sep 09 20:52:09.207310 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.207208    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:52:09.208450 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.208435    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:09.209300 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.209282    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.211501 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.211485    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:52:09.211592 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.211543    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mmtbs\""
Sep 09 20:52:09.211653 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.211617    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.211653 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.211565    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:52:09.213864 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.213849    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.215218 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.215203    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.216143 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216126    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.216223 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.216426 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216407    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:09.216522 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysconfig\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216522 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-kubernetes\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216522 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216476    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysctl-d\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216522 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216496    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysctl-conf\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216523    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kf52q\""
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-sys\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216578    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-tuned\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8d26c6-85a4-4e7e-b20e-885ea2347822-tmp\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh9f\" (UniqueName: \"kubernetes.io/projected/dd8d26c6-85a4-4e7e-b20e-885ea2347822-kube-api-access-9hh9f\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216641    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-systemd\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.216717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216714    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4e9a020-a8af-4c11-8215-5fe31de5cd29-agent-certs\") pod \"konnectivity-agent-l8c65\" (UID: \"d4e9a020-a8af-4c11-8215-5fe31de5cd29\") " pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.217002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-modprobe-d\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.217002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-run\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.217002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9a020-a8af-4c11-8215-5fe31de5cd29-konnectivity-ca\") pod \"konnectivity-agent-l8c65\" (UID: \"d4e9a020-a8af-4c11-8215-5fe31de5cd29\") " pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.217002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216811    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc424\" (UniqueName: \"kubernetes.io/projected/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-kube-api-access-lc424\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:09.217002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-lib-modules\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.217002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-var-lib-kubelet\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.217002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.216859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-host\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.217235 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.217087    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:52:09.217235 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.217161    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.217235 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.217231    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.217337 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.217329    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6k6gn\""
Sep 09 20:52:09.217376 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.217334    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:52:09.218421 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.218406    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.220518 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.220504    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.222927 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.222911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.225212 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.225194    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.225311 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.225262    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:09.225537 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.225515    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.225643 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.225576    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.226501 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.226483    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:52:09.226617 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.226508    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.226842 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.226826    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.228197 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.227589    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:52:09.228197 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.227915    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:52:09.228197 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.228105    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6fcb\""
Sep 09 20:52:09.228197 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.228162    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.228414 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.228265    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:52:09.229040 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.229020    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:52:09.229279 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.229258    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.229454 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.229435    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:52:09.229678 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.229662    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:52:09.229756 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.229700    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:52:09.229959 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.229942    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h24zj\""
Sep 09 20:52:09.230084 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.230065    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qlc2g\""
Sep 09 20:52:09.230142 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.230084    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.231298 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.230855    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:52:09.231298 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.230884    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4s59\""
Sep 09 20:52:09.231438 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.231424    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:52:09.236505 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.236491    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.236952 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.236934    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hvcgc\""
Sep 09 20:52:09.237075 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.237059    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.244865 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.244829    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:47:08 +0000 UTC" deadline="2028-06-24 11:41:34.10386428 +0000 UTC"
Sep 09 20:52:09.244929 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.244864    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15686h49m24.859002482s"
Sep 09 20:52:09.246357 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.246332    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal"]
Sep 09 20:52:09.246655 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.246642    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:09.246707 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.246698    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"
Sep 09 20:52:09.259198 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.259182    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:09.262469 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.262454    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal"]
Sep 09 20:52:09.262524 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.262476    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:52:09.286929 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.286911    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9dlz7"
Sep 09 20:52:09.295639 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.295620    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9dlz7"
Sep 09 20:52:09.309544 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.309523    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:52:09.317831 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-modprobe-d\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.317921 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-cni-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.317921 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317860    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-cni-bin\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.317921 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-kubelet\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.317921 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-var-lib-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.318105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovn-node-metrics-cert\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.318105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.318105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc424\" (UniqueName: \"kubernetes.io/projected/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-kube-api-access-lc424\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:09.318105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317963    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-modprobe-d\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.317994    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-var-lib-kubelet\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:09.318105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-var-lib-kubelet\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318350 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318117    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-cnibin\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.318350 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.318350 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-kubernetes\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318350 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysctl-d\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318350 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-sys\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318350 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh9f\" (UniqueName: \"kubernetes.io/projected/dd8d26c6-85a4-4e7e-b20e-885ea2347822-kube-api-access-9hh9f\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318350 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-kubernetes\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysctl-d\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318383    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-sys\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-system-cni-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-systemd\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-registration-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-systemd\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318489    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-ovn\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8ns\" (UniqueName: \"kubernetes.io/projected/796b5cc7-bfa3-4f0e-ba87-73d14a885180-kube-api-access-mr8ns\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-run-netns\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318547    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-systemd\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-lib-modules\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.318633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318621    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-host\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-netns\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318656    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-multus-certs\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-cni-bin\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318701    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-host\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318709    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cfac31b-51fd-4662-8938-4f217ab90c2d-serviceca\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckb6h\" (UniqueName: \"kubernetes.io/projected/9cfac31b-51fd-4662-8938-4f217ab90c2d-kube-api-access-ckb6h\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/450b2f6d-0830-47a9-93fd-f7a1d0549997-iptables-alerter-script\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4e9a020-a8af-4c11-8215-5fe31de5cd29-agent-certs\") pod \"konnectivity-agent-l8c65\" (UID: \"d4e9a020-a8af-4c11-8215-5fe31de5cd29\") " pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-lib-modules\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-socket-dir-parent\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.318880    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.318934    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:09.818919958 +0000 UTC m=+2.029463745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-cni-multus\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-hostroot\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.318993    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-etc-kubernetes\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319181 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovnkube-config\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/796b5cc7-bfa3-4f0e-ba87-73d14a885180-tmp-dir\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-socket-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319086    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-cnibin\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-node-log\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-log-socket\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2sz\" (UniqueName: \"kubernetes.io/projected/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-kube-api-access-7g2sz\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/450b2f6d-0830-47a9-93fd-f7a1d0549997-host-slash\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-k8s-cni-cncf-io\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-kubelet\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319320    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-system-cni-dir\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysconfig\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-tuned\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8d26c6-85a4-4e7e-b20e-885ea2347822-tmp\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.319892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-conf-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysconfig\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-daemon-config\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-slash\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd6adce-d09e-42db-8a98-edca9cfb2c26-dbus\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cfac31b-51fd-4662-8938-4f217ab90c2d-host\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319508    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-systemd-units\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-run\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9a020-a8af-4c11-8215-5fe31de5cd29-konnectivity-ca\") pod \"konnectivity-agent-l8c65\" (UID: \"d4e9a020-a8af-4c11-8215-5fe31de5cd29\") " pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94f11843-207d-4a2a-89e0-e997b588d1cd-cni-binary-copy\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319636    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-run\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-cni-netd\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319747    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-env-overrides\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319771    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-etc-selinux\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kw7n\" (UniqueName: \"kubernetes.io/projected/94f11843-207d-4a2a-89e0-e997b588d1cd-kube-api-access-7kw7n\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysctl-conf\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-etc-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319897    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovnkube-script-lib\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319930    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bmw\" (UniqueName: \"kubernetes.io/projected/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-kube-api-access-p6bmw\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/796b5cc7-bfa3-4f0e-ba87-73d14a885180-hosts-file\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319958    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-sysctl-conf\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-sys-fs\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.319983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttzx\" (UniqueName: \"kubernetes.io/projected/450b2f6d-0830-47a9-93fd-f7a1d0549997-kube-api-access-zttzx\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.320017    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd6adce-d09e-42db-8a98-edca9cfb2c26-kubelet-config\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.320037    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-device-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.320058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.320072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lzp\" (UniqueName: \"kubernetes.io/projected/f4b4698e-d000-47db-9b23-b583509bcf5f-kube-api-access-w2lzp\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.320141    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4e9a020-a8af-4c11-8215-5fe31de5cd29-konnectivity-ca\") pod \"konnectivity-agent-l8c65\" (UID: \"d4e9a020-a8af-4c11-8215-5fe31de5cd29\") " pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.320935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.320332    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:52:09.323395 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.323373    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8d26c6-85a4-4e7e-b20e-885ea2347822-tmp\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.323481 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.323429    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8d26c6-85a4-4e7e-b20e-885ea2347822-etc-tuned\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.323690 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.323675    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4e9a020-a8af-4c11-8215-5fe31de5cd29-agent-certs\") pod \"konnectivity-agent-l8c65\" (UID: \"d4e9a020-a8af-4c11-8215-5fe31de5cd29\") " pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.326902 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.326883    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:09.326902 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.326900    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:09.327046 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.326910    2573 projected.go:196] Error preparing data for projected volume kube-api-access-298s7 for pod openshift-network-diagnostics/network-check-target-k2kqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:09.327046 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.326957    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7 podName:ae4d02a8-3af7-4d46-85e9-675222272c94 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:09.826944685 +0000 UTC m=+2.037488448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-298s7" (UniqueName: "kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7") pod "network-check-target-k2kqd" (UID: "ae4d02a8-3af7-4d46-85e9-675222272c94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:09.328760 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.328742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh9f\" (UniqueName: \"kubernetes.io/projected/dd8d26c6-85a4-4e7e-b20e-885ea2347822-kube-api-access-9hh9f\") pod \"tuned-fbj89\" (UID: \"dd8d26c6-85a4-4e7e-b20e-885ea2347822\") " pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.329000 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.328982    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc424\" (UniqueName: \"kubernetes.io/projected/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-kube-api-access-lc424\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:09.421138 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-netns\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-multus-certs\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-cni-bin\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421208    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cfac31b-51fd-4662-8938-4f217ab90c2d-serviceca\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421223    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-multus-certs\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-netns\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckb6h\" (UniqueName: \"kubernetes.io/projected/9cfac31b-51fd-4662-8938-4f217ab90c2d-kube-api-access-ckb6h\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.421329 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-cni-bin\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/450b2f6d-0830-47a9-93fd-f7a1d0549997-iptables-alerter-script\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-socket-dir-parent\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-cni-multus\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-hostroot\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-etc-kubernetes\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovnkube-config\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421509    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-socket-dir-parent\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/796b5cc7-bfa3-4f0e-ba87-73d14a885180-tmp-dir\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-hostroot\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421614    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-cni-multus\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-socket-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-etc-kubernetes\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-cnibin\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421712    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-cnibin\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421763    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cfac31b-51fd-4662-8938-4f217ab90c2d-serviceca\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421804    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-socket-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.421834 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-node-log\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-log-socket\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2sz\" (UniqueName: \"kubernetes.io/projected/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-kube-api-access-7g2sz\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/450b2f6d-0830-47a9-93fd-f7a1d0549997-host-slash\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-k8s-cni-cncf-io\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421954    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/450b2f6d-0830-47a9-93fd-f7a1d0549997-iptables-alerter-script\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-kubelet\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422005    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-system-cni-dir\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422028    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-node-log\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422050    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-conf-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-log-socket\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-daemon-config\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422106    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-slash\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.422710 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd6adce-d09e-42db-8a98-edca9cfb2c26-dbus\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422171    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-system-cni-dir\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cfac31b-51fd-4662-8938-4f217ab90c2d-host\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.421983    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-kubelet\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422204    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-systemd-units\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422239    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cd6adce-d09e-42db-8a98-edca9cfb2c26-dbus\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94f11843-207d-4a2a-89e0-e997b588d1cd-cni-binary-copy\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422274    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-cni-netd\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422284    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-run-k8s-cni-cncf-io\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422288    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovnkube-config\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422298    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-env-overrides\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422140    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-slash\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422321    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-conf-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-etc-selinux\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kw7n\" (UniqueName: \"kubernetes.io/projected/94f11843-207d-4a2a-89e0-e997b588d1cd-kube-api-access-7kw7n\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-etc-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-etc-selinux\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.423573 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovnkube-script-lib\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bmw\" (UniqueName: \"kubernetes.io/projected/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-kube-api-access-p6bmw\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422462    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-systemd-units\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422481    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/796b5cc7-bfa3-4f0e-ba87-73d14a885180-hosts-file\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422482    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-cni-netd\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-sys-fs\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttzx\" (UniqueName: \"kubernetes.io/projected/450b2f6d-0830-47a9-93fd-f7a1d0549997-kube-api-access-zttzx\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd6adce-d09e-42db-8a98-edca9cfb2c26-kubelet-config\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-device-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422653    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lzp\" (UniqueName: \"kubernetes.io/projected/f4b4698e-d000-47db-9b23-b583509bcf5f-kube-api-access-w2lzp\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-cni-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-cni-bin\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422725    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-kubelet\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-var-lib-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovn-node-metrics-cert\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422796    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.424362 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-cnibin\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422891    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-system-cni-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422906    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cd6adce-d09e-42db-8a98-edca9cfb2c26-kubelet-config\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-systemd\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422922    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/796b5cc7-bfa3-4f0e-ba87-73d14a885180-tmp-dir\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422945    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94f11843-207d-4a2a-89e0-e997b588d1cd-cni-binary-copy\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-registration-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422960    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-systemd\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422358    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cfac31b-51fd-4662-8938-4f217ab90c2d-host\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422966    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-cni-binary-copy\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-ovn\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-device-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423015    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423020    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-etc-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8ns\" (UniqueName: \"kubernetes.io/projected/796b5cc7-bfa3-4f0e-ba87-73d14a885180-kube-api-access-mr8ns\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423067    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-cnibin\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.423089    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:09.425072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423120    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.423173    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret podName:1cd6adce-d09e-42db-8a98-edca9cfb2c26 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:09.923127486 +0000 UTC m=+2.133671254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret") pod "global-pull-secret-syncer-c45vx" (UID: "1cd6adce-d09e-42db-8a98-edca9cfb2c26") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423225    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-host-var-lib-cni-bin\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4b4698e-d000-47db-9b23-b583509bcf5f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.422859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-env-overrides\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-run-netns\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423398    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/450b2f6d-0830-47a9-93fd-f7a1d0549997-host-slash\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423426    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-var-lib-openvswitch\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-run-netns\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/796b5cc7-bfa3-4f0e-ba87-73d14a885180-hosts-file\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423493    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-daemon-config\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-sys-fs\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423567    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-host-kubelet\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423615    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-registration-dir\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-run-ovn\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-multus-cni-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423872    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94f11843-207d-4a2a-89e0-e997b588d1cd-system-cni-dir\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.425789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423904    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovnkube-script-lib\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.426262 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.423909    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.426262 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.424257    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4b4698e-d000-47db-9b23-b583509bcf5f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.426736 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.426718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-ovn-node-metrics-cert\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.434010 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.433947    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2sz\" (UniqueName: \"kubernetes.io/projected/abe2c4b9-b1e2-4915-ba8d-01ce0abe0837-kube-api-access-7g2sz\") pod \"aws-ebs-csi-driver-node-5xffl\" (UID: \"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.435498 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.435471    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8ns\" (UniqueName: \"kubernetes.io/projected/796b5cc7-bfa3-4f0e-ba87-73d14a885180-kube-api-access-mr8ns\") pod \"node-resolver-dq49d\" (UID: \"796b5cc7-bfa3-4f0e-ba87-73d14a885180\") " pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.436238 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.436199    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kw7n\" (UniqueName: \"kubernetes.io/projected/94f11843-207d-4a2a-89e0-e997b588d1cd-kube-api-access-7kw7n\") pod \"multus-g7mfh\" (UID: \"94f11843-207d-4a2a-89e0-e997b588d1cd\") " pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.436526 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.436507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bmw\" (UniqueName: \"kubernetes.io/projected/4cfab0e2-e12b-4fed-b42a-00bdf21fd385-kube-api-access-p6bmw\") pod \"ovnkube-node-lbk97\" (UID: \"4cfab0e2-e12b-4fed-b42a-00bdf21fd385\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.436652 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.436633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttzx\" (UniqueName: \"kubernetes.io/projected/450b2f6d-0830-47a9-93fd-f7a1d0549997-kube-api-access-zttzx\") pod \"iptables-alerter-vn5xq\" (UID: \"450b2f6d-0830-47a9-93fd-f7a1d0549997\") " pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.436788 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.436764    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lzp\" (UniqueName: \"kubernetes.io/projected/f4b4698e-d000-47db-9b23-b583509bcf5f-kube-api-access-w2lzp\") pod \"multus-additional-cni-plugins-q5fd2\" (UID: \"f4b4698e-d000-47db-9b23-b583509bcf5f\") " pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.437648 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.437629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckb6h\" (UniqueName: \"kubernetes.io/projected/9cfac31b-51fd-4662-8938-4f217ab90c2d-kube-api-access-ckb6h\") pod \"node-ca-xtcgf\" (UID: \"9cfac31b-51fd-4662-8938-4f217ab90c2d\") " pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.528176 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.527996    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c50a8950375d1e1a79c4ab902dfc9e2.slice/crio-cf98af01852281922d89f7c03150d9bd82d61d8e1788701cb078146e4de2aa17 WatchSource:0}: Error finding container cf98af01852281922d89f7c03150d9bd82d61d8e1788701cb078146e4de2aa17: Status 404 returned error can't find the container with id cf98af01852281922d89f7c03150d9bd82d61d8e1788701cb078146e4de2aa17
Sep 09 20:52:09.534597 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.534579    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:52:09.539152 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.539130    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fbj89"
Sep 09 20:52:09.548005 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.547983    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8d26c6_85a4_4e7e_b20e_885ea2347822.slice/crio-43928e2ff4ed58ac4bc95f327a139fc69d7176d7ed7a5770268e92c65e2cd0de WatchSource:0}: Error finding container 43928e2ff4ed58ac4bc95f327a139fc69d7176d7ed7a5770268e92c65e2cd0de: Status 404 returned error can't find the container with id 43928e2ff4ed58ac4bc95f327a139fc69d7176d7ed7a5770268e92c65e2cd0de
Sep 09 20:52:09.555635 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.555585    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:09.565720 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.565698    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl"
Sep 09 20:52:09.567460 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.567435    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e9a020_a8af_4c11_8215_5fe31de5cd29.slice/crio-5a61f8dce93456aca205e314453f7e9a183a85b362af74ee1a3d25f1ec0ab2b2 WatchSource:0}: Error finding container 5a61f8dce93456aca205e314453f7e9a183a85b362af74ee1a3d25f1ec0ab2b2: Status 404 returned error can't find the container with id 5a61f8dce93456aca205e314453f7e9a183a85b362af74ee1a3d25f1ec0ab2b2
Sep 09 20:52:09.567933 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.567913    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e0ae4901bfc17af41e47ecf4fc10e2.slice/crio-4f86db8ebf04704b5c228b4a4b3a5a1e43b9e82582075388442d3e147a0f5681 WatchSource:0}: Error finding container 4f86db8ebf04704b5c228b4a4b3a5a1e43b9e82582075388442d3e147a0f5681: Status 404 returned error can't find the container with id 4f86db8ebf04704b5c228b4a4b3a5a1e43b9e82582075388442d3e147a0f5681
Sep 09 20:52:09.575166 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.575145    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe2c4b9_b1e2_4915_ba8d_01ce0abe0837.slice/crio-428541731e2e2fd480d7861392d609e7fcc69bcff1e7dafb62f777373f997044 WatchSource:0}: Error finding container 428541731e2e2fd480d7861392d609e7fcc69bcff1e7dafb62f777373f997044: Status 404 returned error can't find the container with id 428541731e2e2fd480d7861392d609e7fcc69bcff1e7dafb62f777373f997044
Sep 09 20:52:09.598451 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.598415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xtcgf"
Sep 09 20:52:09.601673 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.601656    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:09.608370 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.608348    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cfac31b_51fd_4662_8938_4f217ab90c2d.slice/crio-20b23f71c96b8cb27670a8c4b14f5cefd9cc0a34b31675bc43446fee58244af3 WatchSource:0}: Error finding container 20b23f71c96b8cb27670a8c4b14f5cefd9cc0a34b31675bc43446fee58244af3: Status 404 returned error can't find the container with id 20b23f71c96b8cb27670a8c4b14f5cefd9cc0a34b31675bc43446fee58244af3
Sep 09 20:52:09.614791 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.614773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q5fd2"
Sep 09 20:52:09.623110 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.623087    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b4698e_d000_47db_9b23_b583509bcf5f.slice/crio-087e1f6694046f42573ad0e02914f14144b90e5477304a55ee3ea64c8dd62db2 WatchSource:0}: Error finding container 087e1f6694046f42573ad0e02914f14144b90e5477304a55ee3ea64c8dd62db2: Status 404 returned error can't find the container with id 087e1f6694046f42573ad0e02914f14144b90e5477304a55ee3ea64c8dd62db2
Sep 09 20:52:09.634155 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.634135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g7mfh"
Sep 09 20:52:09.643124 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.643103    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f11843_207d_4a2a_89e0_e997b588d1cd.slice/crio-e1b874231df94a59ca0a9c7747df9b32292e66d0f13a167fcd323ec2d5830de5 WatchSource:0}: Error finding container e1b874231df94a59ca0a9c7747df9b32292e66d0f13a167fcd323ec2d5830de5: Status 404 returned error can't find the container with id e1b874231df94a59ca0a9c7747df9b32292e66d0f13a167fcd323ec2d5830de5
Sep 09 20:52:09.649933 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.649916    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vn5xq"
Sep 09 20:52:09.658286 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.658263    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450b2f6d_0830_47a9_93fd_f7a1d0549997.slice/crio-eb78014768d6370bfb786ba4b61eeddaff4de1aa1c97e7f283b8e982bc4518af WatchSource:0}: Error finding container eb78014768d6370bfb786ba4b61eeddaff4de1aa1c97e7f283b8e982bc4518af: Status 404 returned error can't find the container with id eb78014768d6370bfb786ba4b61eeddaff4de1aa1c97e7f283b8e982bc4518af
Sep 09 20:52:09.667468 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.667448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:09.675021 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.675001    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dq49d"
Sep 09 20:52:09.677399 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:09.677378    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfab0e2_e12b_4fed_b42a_00bdf21fd385.slice/crio-6f25b1b2aa9f5c56fdcb37c183123ac73df497904d85f957f7d6b2273e3e3be9 WatchSource:0}: Error finding container 6f25b1b2aa9f5c56fdcb37c183123ac73df497904d85f957f7d6b2273e3e3be9: Status 404 returned error can't find the container with id 6f25b1b2aa9f5c56fdcb37c183123ac73df497904d85f957f7d6b2273e3e3be9
Sep 09 20:52:09.685429 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.685410    2573 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-dq49d_openshift-dns(796b5cc7-bfa3-4f0e-ba87-73d14a885180): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 09 20:52:09.685472 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.685448    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-dq49d" podUID="796b5cc7-bfa3-4f0e-ba87-73d14a885180"
Sep 09 20:52:09.826999 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.826923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:09.827188 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.827097    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:09.827188 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.827168    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:10.827151358 +0000 UTC m=+3.037695125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:09.928172 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.928138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:09.928387 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:09.928203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:09.928387 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.928316    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:09.928542 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.928394    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret podName:1cd6adce-d09e-42db-8a98-edca9cfb2c26 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:10.928374335 +0000 UTC m=+3.138918121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret") pod "global-pull-secret-syncer-c45vx" (UID: "1cd6adce-d09e-42db-8a98-edca9cfb2c26") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:09.928542 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.928322    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:09.928542 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.928458    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:09.928542 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.928476    2573 projected.go:196] Error preparing data for projected volume kube-api-access-298s7 for pod openshift-network-diagnostics/network-check-target-k2kqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:09.928542 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:09.928525    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7 podName:ae4d02a8-3af7-4d46-85e9-675222272c94 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:10.928513299 +0000 UTC m=+3.139057063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-298s7" (UniqueName: "kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7") pod "network-check-target-k2kqd" (UID: "ae4d02a8-3af7-4d46-85e9-675222272c94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:10.282980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.282940    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:10.283484 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.283100    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:10.300119 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.300058    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:09 +0000 UTC" deadline="2028-02-05 07:32:34.273775234 +0000 UTC"
Sep 09 20:52:10.300119 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.300116    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12322h40m23.97366323s"
Sep 09 20:52:10.309706 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.309095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"6f25b1b2aa9f5c56fdcb37c183123ac73df497904d85f957f7d6b2273e3e3be9"}
Sep 09 20:52:10.313909 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.313875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vn5xq" event={"ID":"450b2f6d-0830-47a9-93fd-f7a1d0549997","Type":"ContainerStarted","Data":"eb78014768d6370bfb786ba4b61eeddaff4de1aa1c97e7f283b8e982bc4518af"}
Sep 09 20:52:10.316253 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.316221    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g7mfh" event={"ID":"94f11843-207d-4a2a-89e0-e997b588d1cd","Type":"ContainerStarted","Data":"e1b874231df94a59ca0a9c7747df9b32292e66d0f13a167fcd323ec2d5830de5"}
Sep 09 20:52:10.318366 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.318341    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xtcgf" event={"ID":"9cfac31b-51fd-4662-8938-4f217ab90c2d","Type":"ContainerStarted","Data":"20b23f71c96b8cb27670a8c4b14f5cefd9cc0a34b31675bc43446fee58244af3"}
Sep 09 20:52:10.320528 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.320497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl" event={"ID":"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837","Type":"ContainerStarted","Data":"428541731e2e2fd480d7861392d609e7fcc69bcff1e7dafb62f777373f997044"}
Sep 09 20:52:10.340899 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.340861    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:52:10.345682 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.345578    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal" event={"ID":"c5e0ae4901bfc17af41e47ecf4fc10e2","Type":"ContainerStarted","Data":"4f86db8ebf04704b5c228b4a4b3a5a1e43b9e82582075388442d3e147a0f5681"}
Sep 09 20:52:10.349436 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.349361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l8c65" event={"ID":"d4e9a020-a8af-4c11-8215-5fe31de5cd29","Type":"ContainerStarted","Data":"5a61f8dce93456aca205e314453f7e9a183a85b362af74ee1a3d25f1ec0ab2b2"}
Sep 09 20:52:10.353380 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.353317    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerStarted","Data":"087e1f6694046f42573ad0e02914f14144b90e5477304a55ee3ea64c8dd62db2"}
Sep 09 20:52:10.357424 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.357388    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fbj89" event={"ID":"dd8d26c6-85a4-4e7e-b20e-885ea2347822","Type":"ContainerStarted","Data":"43928e2ff4ed58ac4bc95f327a139fc69d7176d7ed7a5770268e92c65e2cd0de"}
Sep 09 20:52:10.359499 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.359472    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal" event={"ID":"1c50a8950375d1e1a79c4ab902dfc9e2","Type":"ContainerStarted","Data":"cf98af01852281922d89f7c03150d9bd82d61d8e1788701cb078146e4de2aa17"}
Sep 09 20:52:10.362938 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.362887    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dq49d" event={"ID":"796b5cc7-bfa3-4f0e-ba87-73d14a885180","Type":"ContainerStarted","Data":"272b0c235f795813ab224e82e9597dbe470fb1915e5c7ee41bee98fc3cb7d660"}
Sep 09 20:52:10.365884 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.365847    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40bc1b9c6b608369978e9f165c5e22d7a75fd91b3480cba60cd275228d721616\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-dq49d" podUID="796b5cc7-bfa3-4f0e-ba87-73d14a885180"
Sep 09 20:52:10.846404 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.846362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:10.846620 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.846527    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:10.846620 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.846614    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:12.846575799 +0000 UTC m=+5.057119579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:10.946809 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.946761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:10.947013 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:10.946828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:10.947013 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.947001    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:10.947127 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.947070    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret podName:1cd6adce-d09e-42db-8a98-edca9cfb2c26 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:12.947049805 +0000 UTC m=+5.157593573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret") pod "global-pull-secret-syncer-c45vx" (UID: "1cd6adce-d09e-42db-8a98-edca9cfb2c26") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:10.947545 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.947519    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:10.947545 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.947548    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:10.947737 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.947562    2573 projected.go:196] Error preparing data for projected volume kube-api-access-298s7 for pod openshift-network-diagnostics/network-check-target-k2kqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:10.947737 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:10.947610    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7 podName:ae4d02a8-3af7-4d46-85e9-675222272c94 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:12.947594833 +0000 UTC m=+5.158138612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-298s7" (UniqueName: "kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7") pod "network-check-target-k2kqd" (UID: "ae4d02a8-3af7-4d46-85e9-675222272c94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:11.283746 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:11.283056    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:11.283746 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:11.283211    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:11.283746 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:11.283056    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:11.283746 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:11.283319    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:11.301225 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:11.301111    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:47:09 +0000 UTC" deadline="2028-03-23 13:29:14.510655496 +0000 UTC"
Sep 09 20:52:11.301225 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:11.301151    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13456h37m3.2095096s"
Sep 09 20:52:12.283091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:12.283051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:12.283261 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.283211    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:12.865576 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:12.865527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:12.866124 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.865723    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:12.866124 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.865793    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:16.865771597 +0000 UTC m=+9.076315377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:12.966403 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:12.966342    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:12.966403 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:12.966383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:12.966655 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.966508    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:12.966655 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.966522    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:12.966655 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.966539    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:12.966655 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.966552    2573 projected.go:196] Error preparing data for projected volume kube-api-access-298s7 for pod openshift-network-diagnostics/network-check-target-k2kqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:12.966655 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.966575    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret podName:1cd6adce-d09e-42db-8a98-edca9cfb2c26 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:16.966553332 +0000 UTC m=+9.177097112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret") pod "global-pull-secret-syncer-c45vx" (UID: "1cd6adce-d09e-42db-8a98-edca9cfb2c26") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:12.966655 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:12.966614    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7 podName:ae4d02a8-3af7-4d46-85e9-675222272c94 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:16.966603092 +0000 UTC m=+9.177146857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-298s7" (UniqueName: "kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7") pod "network-check-target-k2kqd" (UID: "ae4d02a8-3af7-4d46-85e9-675222272c94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:13.282039 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:13.281946    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:13.282220 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:13.282079    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:13.282490 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:13.282468    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:13.282639 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:13.282578    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:14.291782 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:14.291749    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:14.292268 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:14.291894    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:15.282719 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:15.282680    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:15.282918 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:15.282829    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:15.283308 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:15.283285    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:15.283418 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:15.283372    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:16.285840 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:16.285806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:16.286323 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:16.285942    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:16.903942 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:16.903305    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:16.903942 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:16.903496    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:16.903942 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:16.903580    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.903543213 +0000 UTC m=+17.114086980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:17.004770 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:17.004628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:17.004770 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:17.004700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:17.004992 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.004887    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:17.004992 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.004911    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:17.004992 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.004925    2573 projected.go:196] Error preparing data for projected volume kube-api-access-298s7 for pod openshift-network-diagnostics/network-check-target-k2kqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:17.004992 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.004985    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7 podName:ae4d02a8-3af7-4d46-85e9-675222272c94 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:25.004967418 +0000 UTC m=+17.215511182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-298s7" (UniqueName: "kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7") pod "network-check-target-k2kqd" (UID: "ae4d02a8-3af7-4d46-85e9-675222272c94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:17.005475 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.005454    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:17.005592 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.005517    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret podName:1cd6adce-d09e-42db-8a98-edca9cfb2c26 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:25.005502 +0000 UTC m=+17.216045763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret") pod "global-pull-secret-syncer-c45vx" (UID: "1cd6adce-d09e-42db-8a98-edca9cfb2c26") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:17.282147 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:17.282112    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:17.282346 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:17.282159    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:17.282346 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.282262    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:17.282466 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:17.282362    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:18.283251 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:18.283225    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:18.283649 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:18.283318    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:19.282337 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:19.282303    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:19.282519 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:19.282414    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:19.282519 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:19.282445    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:19.282519 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:19.282491    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:20.282968 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:20.282930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:20.283433 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:20.283076    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:21.282453 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:21.282422    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:21.282630 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:21.282431    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:21.282630 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:21.282526    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:21.282707 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:21.282627    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:22.282862 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:22.282820    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:22.283483 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:22.282981    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:23.282499 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:23.282470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:23.282699 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:23.282505    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:23.282699 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:23.282610    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:23.282822 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:23.282746    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:24.282840 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:24.282806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:24.283282 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:24.282950    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:24.966143 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:24.966107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:24.966339 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:24.966230    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:24.966339 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:24.966296    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:40.966277491 +0000 UTC m=+33.176821253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:25.066918 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:25.066889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:25.066918 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:25.066923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:25.067101 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.067019    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:25.067101 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.067048    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:25.067101 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.067064    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret podName:1cd6adce-d09e-42db-8a98-edca9cfb2c26 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:41.067052504 +0000 UTC m=+33.277596266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret") pod "global-pull-secret-syncer-c45vx" (UID: "1cd6adce-d09e-42db-8a98-edca9cfb2c26") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:25.067101 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.067070    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:25.067101 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.067080    2573 projected.go:196] Error preparing data for projected volume kube-api-access-298s7 for pod openshift-network-diagnostics/network-check-target-k2kqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:25.067298 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.067118    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7 podName:ae4d02a8-3af7-4d46-85e9-675222272c94 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:41.067106059 +0000 UTC m=+33.277649839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-298s7" (UniqueName: "kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7") pod "network-check-target-k2kqd" (UID: "ae4d02a8-3af7-4d46-85e9-675222272c94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:25.282097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:25.282016    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:25.282265 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.282132    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:25.282265 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:25.282148    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:25.282382 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:25.282303    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:26.282717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:26.282685    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:26.283242 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:26.282825    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:27.282759 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:27.282713    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:27.282759 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:27.282744    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:27.283356 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:27.282851    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:27.283356 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:27.282934    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:28.283647 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:28.283617    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:28.284099 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:28.283745    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:29.282758 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.282537    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:29.282850 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.282585    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:29.282898 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:29.282869    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:29.282943 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:29.282904    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:29.432850 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.432820    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 20:52:29.434852 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.434824    2573 generic.go:356] "Generic (PLEG): container finished" podID="4cfab0e2-e12b-4fed-b42a-00bdf21fd385" containerID="481280730e954bfa72a3809481c8a27c33e5db2a4d64dbb5a800c5c75fb88016" exitCode=1
Sep 09 20:52:29.434960 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.434914    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"32c9ca0d9653134bdb707279eed85febb79408ecb35d9bde7da2e85cee5328db"}
Sep 09 20:52:29.435016 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.435000    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerDied","Data":"481280730e954bfa72a3809481c8a27c33e5db2a4d64dbb5a800c5c75fb88016"}
Sep 09 20:52:29.435073 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.435056    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"26f06d914004c9c19a0aab4ddb742f04f6610b8d19106414e70ec27cdaeb2a53"}
Sep 09 20:52:29.438525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.438001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal" event={"ID":"c5e0ae4901bfc17af41e47ecf4fc10e2","Type":"ContainerStarted","Data":"68d051d200acc87f81a57a14210086db802ef5fbba5057b0e901d7207a3ef50d"}
Sep 09 20:52:29.445022 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.443415    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fbj89" event={"ID":"dd8d26c6-85a4-4e7e-b20e-885ea2347822","Type":"ContainerStarted","Data":"fd2a67b48dd838c99c1910cd490d4f0d5d0b19bd0c84e6dafffa14cc631c2402"}
Sep 09 20:52:29.472824 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.472447    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fbj89" podStartSLOduration=2.220418793 podStartE2EDuration="21.472424633s" totalImagesPullingTime="19.25200584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:29.471697132 +0000 UTC m=+21.682240954" watchObservedRunningTime="2026-09-09 20:52:29.472424633 +0000 UTC m=+21.682968417"
Sep 09 20:52:29.472991 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:29.472869    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-205.ec2.internal" podStartSLOduration=20.472858006 podStartE2EDuration="20.472858006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:29.455989458 +0000 UTC m=+21.666533243" watchObservedRunningTime="2026-09-09 20:52:29.472858006 +0000 UTC m=+21.683401804"
Sep 09 20:52:30.283044 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.283019    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:30.283180 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:30.283162    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:30.448625 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.448584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dq49d" event={"ID":"796b5cc7-bfa3-4f0e-ba87-73d14a885180","Type":"ContainerStarted","Data":"f24960af77b4632ef6f92587bea8fbc875d42a0ea0a35f78acab65f6bd38fc46"}
Sep 09 20:52:30.454333 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.452356    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 20:52:30.454838 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.454784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"e718a9c990a49e5a76c917cea23d24bc086f8865b81d3025fcfe71baaa013514"}
Sep 09 20:52:30.454938 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.454897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"30a85d8f3b3e4e321e2252d34b5908945800b9100530fd1aac3cf2f36879df5f"}
Sep 09 20:52:30.454989 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.454949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"c42b307e74225a8388856073d05209d21ee3bbd5b1023643f4cfdf84151c1d87"}
Sep 09 20:52:30.456284 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.456259    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vn5xq" event={"ID":"450b2f6d-0830-47a9-93fd-f7a1d0549997","Type":"ContainerStarted","Data":"2bfbddf589f29c87bbb85da7818c39f9263143861c0c9ee2702f69598a4d0251"}
Sep 09 20:52:30.457640 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.457611    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g7mfh" event={"ID":"94f11843-207d-4a2a-89e0-e997b588d1cd","Type":"ContainerStarted","Data":"85d487df77c9104506d08ebb49d585042a8ae945821e7064f009b5f1a8b7a1a4"}
Sep 09 20:52:30.458937 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.458917    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xtcgf" event={"ID":"9cfac31b-51fd-4662-8938-4f217ab90c2d","Type":"ContainerStarted","Data":"1b37559fab77471b46104273127165d9f234379f443451c279242c1648e922ad"}
Sep 09 20:52:30.460314 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.460296    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl" event={"ID":"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837","Type":"ContainerStarted","Data":"dac628caf397600e0a8726ed7386dbe2f34bf2d947d6d2e3f142951f17c9084e"}
Sep 09 20:52:30.461598 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.461577    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l8c65" event={"ID":"d4e9a020-a8af-4c11-8215-5fe31de5cd29","Type":"ContainerStarted","Data":"b1e26e77d02b47c76c780f8d95c1bbd868ebc372c1395967a0d08f815003be45"}
Sep 09 20:52:30.463099 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.463078    2573 generic.go:356] "Generic (PLEG): container finished" podID="f4b4698e-d000-47db-9b23-b583509bcf5f" containerID="ee3cfde5cd118b514d0b172c3948265e345baade587e3d36600244664fbc28ee" exitCode=0
Sep 09 20:52:30.463188 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.463141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerDied","Data":"ee3cfde5cd118b514d0b172c3948265e345baade587e3d36600244664fbc28ee"}
Sep 09 20:52:30.464053 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.464005    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dq49d" podStartSLOduration=21.463986992 podStartE2EDuration="21.463986992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-09 20:52:30.46366936 +0000 UTC m=+22.674213156" watchObservedRunningTime="2026-09-09 20:52:30.463986992 +0000 UTC m=+22.674530914"
Sep 09 20:52:30.464783 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.464764    2573 generic.go:356] "Generic (PLEG): container finished" podID="1c50a8950375d1e1a79c4ab902dfc9e2" containerID="707ebbb3f4e02bfa4791184ffe786bd49920b4071caaadcb9c30fce5552e36e7" exitCode=0
Sep 09 20:52:30.464867 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.464847    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal" event={"ID":"1c50a8950375d1e1a79c4ab902dfc9e2","Type":"ContainerDied","Data":"707ebbb3f4e02bfa4791184ffe786bd49920b4071caaadcb9c30fce5552e36e7"}
Sep 09 20:52:30.480769 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.480728    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g7mfh" podStartSLOduration=1.778505799 podStartE2EDuration="21.480712013s" totalImagesPullingTime="19.702206214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:30.480413991 +0000 UTC m=+22.690957775" watchObservedRunningTime="2026-09-09 20:52:30.480712013 +0000 UTC m=+22.691255797"
Sep 09 20:52:30.502018 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.501974    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vn5xq" podStartSLOduration=2.333432558 podStartE2EDuration="21.50195777s" totalImagesPullingTime="19.168525212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:30.501913441 +0000 UTC m=+22.712457224" watchObservedRunningTime="2026-09-09 20:52:30.50195777 +0000 UTC m=+22.712501554"
Sep 09 20:52:30.542355 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.542311    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xtcgf" podStartSLOduration=2.291850259 podStartE2EDuration="21.542295741s" totalImagesPullingTime="19.250445482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:30.517836899 +0000 UTC m=+22.728380684" watchObservedRunningTime="2026-09-09 20:52:30.542295741 +0000 UTC m=+22.752839524"
Sep 09 20:52:30.568106 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.568046    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l8c65" podStartSLOduration=2.331308145 podStartE2EDuration="21.568026058s" totalImagesPullingTime="19.236717913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:30.567731242 +0000 UTC m=+22.778275026" watchObservedRunningTime="2026-09-09 20:52:30.568026058 +0000 UTC m=+22.778569843"
Sep 09 20:52:30.578713 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:30.578689    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:52:31.282719 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.282683    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:31.282879 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:31.282785    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:31.282879 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.282811    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:31.282879 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:31.282868    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:31.292180 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.292104    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:52:30.578824802Z","UUID":"e462692f-aedc-4138-a6f3-71f92eb896c9","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:52:31.293434 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.293420    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:52:31.293513 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.293444    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:52:31.468920 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.468881    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl" event={"ID":"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837","Type":"ContainerStarted","Data":"11244ca2fe39a976e185012d9411b91e6ab4c9b86261b7b470a430dee6eae2ec"}
Sep 09 20:52:31.470661 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.470624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal" event={"ID":"1c50a8950375d1e1a79c4ab902dfc9e2","Type":"ContainerStarted","Data":"1a7fc3f63a2c355200c1d52c4b4ab09875a8648d6aafa6b1b46b3f23fcfd922f"}
Sep 09 20:52:31.485159 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.485114    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-205.ec2.internal" podStartSLOduration=22.485102341 podStartE2EDuration="22.485102341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:31.484764299 +0000 UTC m=+23.695308084" watchObservedRunningTime="2026-09-09 20:52:31.485102341 +0000 UTC m=+23.695646124"
Sep 09 20:52:31.723783 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.723296    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:31.724623 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:31.724446    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:32.116700 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:32.116457    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:32.117916 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:32.117890    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l8c65"
Sep 09 20:52:32.282404 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:32.282294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:32.282626 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:32.282419    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:32.477090 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:32.477061    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 20:52:32.477663 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:32.477637    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"ffe14a99af1a4f2daed0197ae4a5be7ba4f0e1c140353fc2fe868f832511dd9f"}
Sep 09 20:52:32.480067 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:32.480036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl" event={"ID":"abe2c4b9-b1e2-4915-ba8d-01ce0abe0837","Type":"ContainerStarted","Data":"c524c65f664c86b2030160fca03424934aaaa54e0633da8227368fc93651af54"}
Sep 09 20:52:32.497209 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:32.497156    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5xffl" podStartSLOduration=1.8695657639999999 podStartE2EDuration="23.497138755s" totalImagesPullingTime="21.627572991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:32.496705507 +0000 UTC m=+24.707249308" watchObservedRunningTime="2026-09-09 20:52:32.497138755 +0000 UTC m=+24.707682549"
Sep 09 20:52:33.282295 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:33.282258    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:33.282295 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:33.282291    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:33.282540 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:33.282381    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:33.282540 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:33.282504    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:34.282085 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:34.282049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:34.282487 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:34.282179    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:35.282175 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:35.282142    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:35.282175 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:35.282181    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:35.282660 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:35.282269    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:35.282660 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:35.282378    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:36.282582 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.282330    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:36.282950 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:36.282633    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:36.492504 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.492477    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 20:52:36.492879 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.492855    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"5018ccf8f3879951947004ec04cf18eb033ab844bcb2aff3252004d50f2da4a8"}
Sep 09 20:52:36.493210 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.493091    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:36.493210 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.493158    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:36.493325 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.493296    2573 scope.go:117] "RemoveContainer" containerID="481280730e954bfa72a3809481c8a27c33e5db2a4d64dbb5a800c5c75fb88016"
Sep 09 20:52:36.494948 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.494921    2573 generic.go:356] "Generic (PLEG): container finished" podID="f4b4698e-d000-47db-9b23-b583509bcf5f" containerID="2bce1d0a819f9cc2ba6439ada4aed49bd5221db425e6e64b3b60ce49804a9035" exitCode=0
Sep 09 20:52:36.495058 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.494961    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerDied","Data":"2bce1d0a819f9cc2ba6439ada4aed49bd5221db425e6e64b3b60ce49804a9035"}
Sep 09 20:52:36.509243 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:36.509225    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:37.282081 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.282045    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:37.282270 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.282082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:37.282270 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:37.282193    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:37.282404 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:37.282297    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:37.500335 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.500308    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 20:52:37.501093 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.500712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" event={"ID":"4cfab0e2-e12b-4fed-b42a-00bdf21fd385","Type":"ContainerStarted","Data":"37223fef880ed2dc0a71b2529e0243c37f2d0e6476f7f8091386c9ecd835ff9b"}
Sep 09 20:52:37.501093 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.500925    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:37.502929 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.502909    2573 generic.go:356] "Generic (PLEG): container finished" podID="f4b4698e-d000-47db-9b23-b583509bcf5f" containerID="3bb45d238d2da6b1eafd27dbfa6ce6793c6189440532db61bc30910a37475d90" exitCode=0
Sep 09 20:52:37.503053 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.502945    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerDied","Data":"3bb45d238d2da6b1eafd27dbfa6ce6793c6189440532db61bc30910a37475d90"}
Sep 09 20:52:37.517249 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.517228    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:52:37.582569 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.582506    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97" podStartSLOduration=9.369077051 podStartE2EDuration="28.582491295s" totalImagesPullingTime="19.213414244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:37.547290807 +0000 UTC m=+29.757834590" watchObservedRunningTime="2026-09-09 20:52:37.582491295 +0000 UTC m=+29.793035079"
Sep 09 20:52:37.647956 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.647929    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k2kqd"]
Sep 09 20:52:37.648138 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.648044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:37.648201 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:37.648145    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:37.650910 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.650871    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xr4nz"]
Sep 09 20:52:37.651034 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.650983    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:37.651112 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:37.651091    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:37.651425 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.651409    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c45vx"]
Sep 09 20:52:37.651506 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:37.651472    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:37.651599 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:37.651587    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:38.508508 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:38.508435    2573 generic.go:356] "Generic (PLEG): container finished" podID="f4b4698e-d000-47db-9b23-b583509bcf5f" containerID="e162724f8b230a4ff96ffa4631e021a6171fc936987d0fc56c3d5ae0086132c7" exitCode=0
Sep 09 20:52:38.508952 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:38.508513    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerDied","Data":"e162724f8b230a4ff96ffa4631e021a6171fc936987d0fc56c3d5ae0086132c7"}
Sep 09 20:52:39.282417 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:39.282343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:39.282582 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:39.282343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:39.282582 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:39.282464    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:39.282582 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:39.282348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:39.282582 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:39.282529    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:39.282731 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:39.282647    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:40.980856 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:40.980603    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:40.981325 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:40.980800    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:40.981325 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:40.980966    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:12.980943587 +0000 UTC m=+65.191487350 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:41.082059 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:41.082019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:41.082059 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:41.082060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:41.082301 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.082178    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:41.082301 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.082208    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:41.082301 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.082233    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:41.082301 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.082239    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret podName:1cd6adce-d09e-42db-8a98-edca9cfb2c26 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.082220087 +0000 UTC m=+65.292763852 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret") pod "global-pull-secret-syncer-c45vx" (UID: "1cd6adce-d09e-42db-8a98-edca9cfb2c26") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:41.082301 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.082247    2573 projected.go:196] Error preparing data for projected volume kube-api-access-298s7 for pod openshift-network-diagnostics/network-check-target-k2kqd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:41.082301 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.082299    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7 podName:ae4d02a8-3af7-4d46-85e9-675222272c94 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.082282541 +0000 UTC m=+65.292826305 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-298s7" (UniqueName: "kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7") pod "network-check-target-k2kqd" (UID: "ae4d02a8-3af7-4d46-85e9-675222272c94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:41.282419 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:41.282339    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:41.282598 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:41.282339    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:41.282598 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.282447    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c45vx" podUID="1cd6adce-d09e-42db-8a98-edca9cfb2c26"
Sep 09 20:52:41.282598 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:41.282339    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:41.282598 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.282525    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xr4nz" podUID="6d4bd7ab-dd8f-45d6-a6bc-23642e40c927"
Sep 09 20:52:41.282762 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:41.282623    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k2kqd" podUID="ae4d02a8-3af7-4d46-85e9-675222272c94"
Sep 09 20:52:42.137232 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.137159    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-205.ec2.internal" event="NodeReady"
Sep 09 20:52:42.137658 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.137300    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:52:42.178806 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.178781    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55bc7d5dcd-s4c89"]
Sep 09 20:52:42.209456 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.209425    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bc7d5dcd-s4c89"]
Sep 09 20:52:42.209651 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.209544    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.209718 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.209544    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7fppx"]
Sep 09 20:52:42.212732 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.212584    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:52:42.212732 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.212614    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:52:42.212732 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.212593    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:52:42.212732 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.212636    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lr6q6\""
Sep 09 20:52:42.219418 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.218963    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:52:42.227742 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.227718    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-scgvd"]
Sep 09 20:52:42.228816 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.228797    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.232991 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.231952    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:52:42.232991 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.231952    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:52:42.232991 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.232211    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vdcc\""
Sep 09 20:52:42.240571 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.240537    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7fppx"]
Sep 09 20:52:42.240668 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.240599    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-scgvd"]
Sep 09 20:52:42.240668 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.240643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:42.243159 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.243141    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:52:42.243416 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.243399    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:52:42.243416 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.243406    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:52:42.243693 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.243679    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f6qjq\""
Sep 09 20:52:42.393059 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.392978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-bound-sa-token\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393059 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7177e33-d467-4ed6-bbf5-9687e57c09e4-cert\") pod \"ingress-canary-scgvd\" (UID: \"c7177e33-d467-4ed6-bbf5-9687e57c09e4\") " pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:42.393059 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393055    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-image-registry-private-configuration\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-trusted-ca\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-installation-pull-secrets\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndxv\" (UniqueName: \"kubernetes.io/projected/c7177e33-d467-4ed6-bbf5-9687e57c09e4-kube-api-access-6ndxv\") pod \"ingress-canary-scgvd\" (UID: \"c7177e33-d467-4ed6-bbf5-9687e57c09e4\") " pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:42.393303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393171    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7s7\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-kube-api-access-tg7s7\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-tmp-dir\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.393533 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-config-volume\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.393533 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.393533 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b7fab98-bbd0-48fd-aa24-384491b0a589-ca-trust-extracted\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393533 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-certificates\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.393533 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.393462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbpc\" (UniqueName: \"kubernetes.io/projected/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-kube-api-access-rhbpc\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.494343 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7s7\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-kube-api-access-tg7s7\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-tmp-dir\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-config-volume\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b7fab98-bbd0-48fd-aa24-384491b0a589-ca-trust-extracted\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:42.494488    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-certificates\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:42.494511    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc7d5dcd-s4c89: secret "image-registry-tls" not found
Sep 09 20:52:42.494525 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbpc\" (UniqueName: \"kubernetes.io/projected/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-kube-api-access-rhbpc\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:42.494594    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls podName:1b7fab98-bbd0-48fd-aa24-384491b0a589 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:42.99457107 +0000 UTC m=+35.205114847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls") pod "image-registry-55bc7d5dcd-s4c89" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589") : secret "image-registry-tls" not found
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494654    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-bound-sa-token\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7177e33-d467-4ed6-bbf5-9687e57c09e4-cert\") pod \"ingress-canary-scgvd\" (UID: \"c7177e33-d467-4ed6-bbf5-9687e57c09e4\") " pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-image-registry-private-configuration\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-trusted-ca\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-installation-pull-secrets\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndxv\" (UniqueName: \"kubernetes.io/projected/c7177e33-d467-4ed6-bbf5-9687e57c09e4-kube-api-access-6ndxv\") pod \"ingress-canary-scgvd\" (UID: \"c7177e33-d467-4ed6-bbf5-9687e57c09e4\") " pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:42.494980 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.494807    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-tmp-dir\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.495343 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.495254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-config-volume\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.495343 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:42.495317    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:42.495433 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:42.495353    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls podName:dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:42.995340259 +0000 UTC m=+35.205884022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls") pod "dns-default-7fppx" (UID: "dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:42.495604 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.495579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b7fab98-bbd0-48fd-aa24-384491b0a589-ca-trust-extracted\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.496136 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.496111    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-certificates\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.496392 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.496371    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-trusted-ca\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.499294 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.499270    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-installation-pull-secrets\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.499294 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.499288    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-image-registry-private-configuration\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.499439 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.499302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7177e33-d467-4ed6-bbf5-9687e57c09e4-cert\") pod \"ingress-canary-scgvd\" (UID: \"c7177e33-d467-4ed6-bbf5-9687e57c09e4\") " pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:42.503784 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.503763    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7s7\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-kube-api-access-tg7s7\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.504021 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.504001    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndxv\" (UniqueName: \"kubernetes.io/projected/c7177e33-d467-4ed6-bbf5-9687e57c09e4-kube-api-access-6ndxv\") pod \"ingress-canary-scgvd\" (UID: \"c7177e33-d467-4ed6-bbf5-9687e57c09e4\") " pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:42.504867 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.504845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbpc\" (UniqueName: \"kubernetes.io/projected/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-kube-api-access-rhbpc\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:42.505070 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.505041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-bound-sa-token\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:42.550475 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:42.550419    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-scgvd"
Sep 09 20:52:43.000141 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.000106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:43.000141 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.000142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:43.000406 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:43.000243    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:43.000406 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:43.000267    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:43.000406 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:43.000289    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc7d5dcd-s4c89: secret "image-registry-tls" not found
Sep 09 20:52:43.000406 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:43.000306    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls podName:dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:44.00029182 +0000 UTC m=+36.210835599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls") pod "dns-default-7fppx" (UID: "dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:43.000406 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:43.000346    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls podName:1b7fab98-bbd0-48fd-aa24-384491b0a589 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:44.000327399 +0000 UTC m=+36.210871185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls") pod "image-registry-55bc7d5dcd-s4c89" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589") : secret "image-registry-tls" not found
Sep 09 20:52:43.177926 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.177760    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-scgvd"]
Sep 09 20:52:43.178495 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:43.178473    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7177e33_d467_4ed6_bbf5_9687e57c09e4.slice/crio-04dbcbf3d3634d86ef0251fa437ba9cb151593f593cb84a472969d108634b00c WatchSource:0}: Error finding container 04dbcbf3d3634d86ef0251fa437ba9cb151593f593cb84a472969d108634b00c: Status 404 returned error can't find the container with id 04dbcbf3d3634d86ef0251fa437ba9cb151593f593cb84a472969d108634b00c
Sep 09 20:52:43.282437 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.282414    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:52:43.282590 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.282529    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:52:43.282656 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.282593    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:52:43.285415 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.285391    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9kf4\""
Sep 09 20:52:43.285510 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.285474    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:52:43.286199 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.286181    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:52:43.286310 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.286202    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:52:43.286310 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.286295    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7m4wj\""
Sep 09 20:52:43.286415 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.286309    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:52:43.523564 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.523503    2573 generic.go:356] "Generic (PLEG): container finished" podID="f4b4698e-d000-47db-9b23-b583509bcf5f" containerID="8de34a119b9d682ca0f926d1abcd6bc31b4145372afd76a6b6531856149d0752" exitCode=0
Sep 09 20:52:43.523771 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.523588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerDied","Data":"8de34a119b9d682ca0f926d1abcd6bc31b4145372afd76a6b6531856149d0752"}
Sep 09 20:52:43.524965 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:43.524934    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-scgvd" event={"ID":"c7177e33-d467-4ed6-bbf5-9687e57c09e4","Type":"ContainerStarted","Data":"04dbcbf3d3634d86ef0251fa437ba9cb151593f593cb84a472969d108634b00c"}
Sep 09 20:52:44.009089 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:44.009045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:44.009357 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:44.009101    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:44.009357 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:44.009219    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:44.009357 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:44.009235    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:44.009357 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:44.009237    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc7d5dcd-s4c89: secret "image-registry-tls" not found
Sep 09 20:52:44.009357 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:44.009288    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls podName:dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:46.009273237 +0000 UTC m=+38.219817000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls") pod "dns-default-7fppx" (UID: "dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:44.009357 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:44.009305    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls podName:1b7fab98-bbd0-48fd-aa24-384491b0a589 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:46.009295655 +0000 UTC m=+38.219839417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls") pod "image-registry-55bc7d5dcd-s4c89" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589") : secret "image-registry-tls" not found
Sep 09 20:52:44.530392 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:44.530361    2573 generic.go:356] "Generic (PLEG): container finished" podID="f4b4698e-d000-47db-9b23-b583509bcf5f" containerID="f5f04cede40844bbc20c3769a98ff28212402e76443f2fc9417839e5d5bdec30" exitCode=0
Sep 09 20:52:44.530795 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:44.530402    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerDied","Data":"f5f04cede40844bbc20c3769a98ff28212402e76443f2fc9417839e5d5bdec30"}
Sep 09 20:52:45.536113 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:45.536072    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" event={"ID":"f4b4698e-d000-47db-9b23-b583509bcf5f","Type":"ContainerStarted","Data":"b9389762e870cc61fd3707a40eb4c72c1b4917421a23a72c91861628597cddaa"}
Sep 09 20:52:45.565070 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:45.565017    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q5fd2" podStartSLOduration=5.902984745 podStartE2EDuration="36.564998303s" totalImagesPullingTime="30.662013558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:45.563802885 +0000 UTC m=+37.774346681" watchObservedRunningTime="2026-09-09 20:52:45.564998303 +0000 UTC m=+37.775542086"
Sep 09 20:52:46.025043 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:46.025003    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:46.025043 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:46.025043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:46.025243 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:46.025156    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:46.025243 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:46.025177    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc7d5dcd-s4c89: secret "image-registry-tls" not found
Sep 09 20:52:46.025243 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:46.025185    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:46.025243 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:46.025228    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls podName:dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.025215207 +0000 UTC m=+42.235758969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls") pod "dns-default-7fppx" (UID: "dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:46.025243 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:46.025240    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls podName:1b7fab98-bbd0-48fd-aa24-384491b0a589 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.025234988 +0000 UTC m=+42.235778749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls") pod "image-registry-55bc7d5dcd-s4c89" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589") : secret "image-registry-tls" not found
Sep 09 20:52:46.539700 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:46.539662    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-scgvd" event={"ID":"c7177e33-d467-4ed6-bbf5-9687e57c09e4","Type":"ContainerStarted","Data":"86e112649fddb50df54ab43cfa18a8f9b4eabc0c57cb018406cabe8ded42821a"}
Sep 09 20:52:46.558073 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:46.558012    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-scgvd" podStartSLOduration=2.056555446 podStartE2EDuration="4.55799127s" totalImagesPullingTime="2.501435824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:46.557248235 +0000 UTC m=+38.767792020" watchObservedRunningTime="2026-09-09 20:52:46.55799127 +0000 UTC m=+38.768535054"
Sep 09 20:52:50.055789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:50.055717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:50.055789 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:50.055796    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:50.056249 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:50.055898    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:50.056249 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:50.055908    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:50.056249 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:50.055922    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc7d5dcd-s4c89: secret "image-registry-tls" not found
Sep 09 20:52:50.056249 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:50.055963    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls podName:dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.05594942 +0000 UTC m=+50.266493181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls") pod "dns-default-7fppx" (UID: "dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:50.056249 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:50.055987    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls podName:1b7fab98-bbd0-48fd-aa24-384491b0a589 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.055969457 +0000 UTC m=+50.266513238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls") pod "image-registry-55bc7d5dcd-s4c89" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589") : secret "image-registry-tls" not found
Sep 09 20:52:58.014073 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.013939    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-2qt7h"]
Sep 09 20:52:58.026397 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.026366    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"]
Sep 09 20:52:58.026569 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.026487    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.029105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.029081    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 20:52:58.029437 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.029421    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qmw8l\""
Sep 09 20:52:58.029531 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.029497    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 20:52:58.032271 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.032242    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:52:58.032676 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.032660    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:52:58.037869 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.037846    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 20:52:58.042476 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.042457    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"]
Sep 09 20:52:58.042633 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.042617    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.045403 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.045374    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:58.045504 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.045386    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:58.045504 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.045491    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 20:52:58.045657 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.045586    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qgf79\""
Sep 09 20:52:58.045809 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.045713    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 20:52:58.057428 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.057407    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"]
Sep 09 20:52:58.057526 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.057444    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"]
Sep 09 20:52:58.057526 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.057460    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-2qt7h"]
Sep 09 20:52:58.057625 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.057570    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:58.060563 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.060530    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:58.061131 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.061115    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:58.061202 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.061135    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-thljz\""
Sep 09 20:52:58.061202 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.061120    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 20:52:58.106387 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.106356    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"]
Sep 09 20:52:58.112717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-kube-api-access-8lqz7\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:58.112857 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112742    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/27d62ac1-f632-4428-adf5-25c91d1b8364-snapshots\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.112857 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzhx\" (UniqueName: \"kubernetes.io/projected/27d62ac1-f632-4428-adf5-25c91d1b8364-kube-api-access-qmzhx\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.112857 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:52:58.112966 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d62ac1-f632-4428-adf5-25c91d1b8364-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.112966 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nt5\" (UniqueName: \"kubernetes.io/projected/b3434296-b3eb-48ef-b19f-95ce0c894107-kube-api-access-t9nt5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.112966 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27d62ac1-f632-4428-adf5-25c91d1b8364-serving-cert\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.112966 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112919    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:52:58.112966 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112942    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d62ac1-f632-4428-adf5-25c91d1b8364-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.112966 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.112978    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.112995    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc7d5dcd-s4c89: secret "image-registry-tls" not found
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.112980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3434296-b3eb-48ef-b19f-95ce0c894107-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.113016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3434296-b3eb-48ef-b19f-95ce0c894107-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.113047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d62ac1-f632-4428-adf5-25c91d1b8364-tmp\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.113053    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.113071    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls podName:1b7fab98-bbd0-48fd-aa24-384491b0a589 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:14.113055482 +0000 UTC m=+66.323599244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls") pod "image-registry-55bc7d5dcd-s4c89" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589") : secret "image-registry-tls" not found
Sep 09 20:52:58.113209 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.113118    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls podName:dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:14.113100306 +0000 UTC m=+66.323644083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls") pod "dns-default-7fppx" (UID: "dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:58.123585 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.123541    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.126676 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.126660    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 20:52:58.127121 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.127106    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:58.128078 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.128053    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 20:52:58.128166 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.128085    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:58.128949 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.128931    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gwk5t\""
Sep 09 20:52:58.129843 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.129790    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"]
Sep 09 20:52:58.213735 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d62ac1-f632-4428-adf5-25c91d1b8364-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.213735 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nt5\" (UniqueName: \"kubernetes.io/projected/b3434296-b3eb-48ef-b19f-95ce0c894107-kube-api-access-t9nt5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27d62ac1-f632-4428-adf5-25c91d1b8364-serving-cert\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d62ac1-f632-4428-adf5-25c91d1b8364-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213792    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3434296-b3eb-48ef-b19f-95ce0c894107-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.213868    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3434296-b3eb-48ef-b19f-95ce0c894107-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.213935    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls podName:7b39b28e-07ba-4e0c-aa16-87d9b83d5a64 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.713916807 +0000 UTC m=+50.924460586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6srvb" (UID: "7b39b28e-07ba-4e0c-aa16-87d9b83d5a64") : secret "samples-operator-tls" not found
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213954    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-config\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.213995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d62ac1-f632-4428-adf5-25c91d1b8364-tmp\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214047 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214035    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-kube-api-access-8lqz7\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:58.214538 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crx4r\" (UniqueName: \"kubernetes.io/projected/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-kube-api-access-crx4r\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.214538 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/27d62ac1-f632-4428-adf5-25c91d1b8364-snapshots\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214538 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-serving-cert\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.214538 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzhx\" (UniqueName: \"kubernetes.io/projected/27d62ac1-f632-4428-adf5-25c91d1b8364-kube-api-access-qmzhx\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214538 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214461    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d62ac1-f632-4428-adf5-25c91d1b8364-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214538 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214513    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3434296-b3eb-48ef-b19f-95ce0c894107-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.214762 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214723    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27d62ac1-f632-4428-adf5-25c91d1b8364-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214835 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214816    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d62ac1-f632-4428-adf5-25c91d1b8364-tmp\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.214881 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.214865    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/27d62ac1-f632-4428-adf5-25c91d1b8364-snapshots\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.217244 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.217223    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3434296-b3eb-48ef-b19f-95ce0c894107-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.217353 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.217303    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27d62ac1-f632-4428-adf5-25c91d1b8364-serving-cert\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.229133 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.229102    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nt5\" (UniqueName: \"kubernetes.io/projected/b3434296-b3eb-48ef-b19f-95ce0c894107-kube-api-access-t9nt5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-dgrqc\" (UID: \"b3434296-b3eb-48ef-b19f-95ce0c894107\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.229133 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.229120    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzhx\" (UniqueName: \"kubernetes.io/projected/27d62ac1-f632-4428-adf5-25c91d1b8364-kube-api-access-qmzhx\") pod \"insights-operator-76bcd6ccf5-2qt7h\" (UID: \"27d62ac1-f632-4428-adf5-25c91d1b8364\") " pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.229494 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.229474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqz7\" (UniqueName: \"kubernetes.io/projected/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-kube-api-access-8lqz7\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:58.314673 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.314645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-config\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.314849 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.314702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crx4r\" (UniqueName: \"kubernetes.io/projected/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-kube-api-access-crx4r\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.314849 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.314723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-serving-cert\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.315238 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.315218    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-config\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.317060 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.317039    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-serving-cert\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.322836 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.322817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crx4r\" (UniqueName: \"kubernetes.io/projected/b1e0d315-b0c6-4e02-9684-b75d86e5b0ff-kube-api-access-crx4r\") pod \"service-ca-operator-859f94c4b-z4254\" (UID: \"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.335780 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.335761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h"
Sep 09 20:52:58.350345 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.350313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"
Sep 09 20:52:58.432971 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.432942    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"
Sep 09 20:52:58.643309 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.643235    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-2qt7h"]
Sep 09 20:52:58.646102 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:58.646075    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d62ac1_f632_4428_adf5_25c91d1b8364.slice/crio-981f202554e91da6267d5f1d5ed80020121a1588c3c39c0b19b418b53961f9be WatchSource:0}: Error finding container 981f202554e91da6267d5f1d5ed80020121a1588c3c39c0b19b418b53961f9be: Status 404 returned error can't find the container with id 981f202554e91da6267d5f1d5ed80020121a1588c3c39c0b19b418b53961f9be
Sep 09 20:52:58.680803 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.680765    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc"]
Sep 09 20:52:58.683703 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:58.683681    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3434296_b3eb_48ef_b19f_95ce0c894107.slice/crio-553b5ea301164c9cb64347a032ff61be0932ef58174afcc3ead1f58e94d34f95 WatchSource:0}: Error finding container 553b5ea301164c9cb64347a032ff61be0932ef58174afcc3ead1f58e94d34f95: Status 404 returned error can't find the container with id 553b5ea301164c9cb64347a032ff61be0932ef58174afcc3ead1f58e94d34f95
Sep 09 20:52:58.718291 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.718256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:58.718461 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.718409    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:58.718504 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:58.718469    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls podName:7b39b28e-07ba-4e0c-aa16-87d9b83d5a64 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:59.718454233 +0000 UTC m=+51.928997998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6srvb" (UID: "7b39b28e-07ba-4e0c-aa16-87d9b83d5a64") : secret "samples-operator-tls" not found
Sep 09 20:52:58.731963 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:52:58.731893    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e0d315_b0c6_4e02_9684_b75d86e5b0ff.slice/crio-67dbf0ce4c2df7889649104e0c18efda7a838c1a66dfe6df93442ac13f9f6282 WatchSource:0}: Error finding container 67dbf0ce4c2df7889649104e0c18efda7a838c1a66dfe6df93442ac13f9f6282: Status 404 returned error can't find the container with id 67dbf0ce4c2df7889649104e0c18efda7a838c1a66dfe6df93442ac13f9f6282
Sep 09 20:52:58.732959 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:58.732931    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254"]
Sep 09 20:52:59.574566 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:59.574505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc" event={"ID":"b3434296-b3eb-48ef-b19f-95ce0c894107","Type":"ContainerStarted","Data":"553b5ea301164c9cb64347a032ff61be0932ef58174afcc3ead1f58e94d34f95"}
Sep 09 20:52:59.580764 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:59.580690    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h" event={"ID":"27d62ac1-f632-4428-adf5-25c91d1b8364","Type":"ContainerStarted","Data":"981f202554e91da6267d5f1d5ed80020121a1588c3c39c0b19b418b53961f9be"}
Sep 09 20:52:59.582591 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:59.582506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254" event={"ID":"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff","Type":"ContainerStarted","Data":"67dbf0ce4c2df7889649104e0c18efda7a838c1a66dfe6df93442ac13f9f6282"}
Sep 09 20:52:59.725905 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:52:59.725869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:52:59.726094 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:59.726036    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:52:59.726158 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:52:59.726099    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls podName:7b39b28e-07ba-4e0c-aa16-87d9b83d5a64 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.726081494 +0000 UTC m=+53.936625272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6srvb" (UID: "7b39b28e-07ba-4e0c-aa16-87d9b83d5a64") : secret "samples-operator-tls" not found
Sep 09 20:53:01.742193 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:01.742148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:53:01.742690 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:01.742338    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:53:01.742690 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:01.742423    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls podName:7b39b28e-07ba-4e0c-aa16-87d9b83d5a64 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.742402693 +0000 UTC m=+57.952946457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6srvb" (UID: "7b39b28e-07ba-4e0c-aa16-87d9b83d5a64") : secret "samples-operator-tls" not found
Sep 09 20:53:02.598568 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:02.598518    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254" event={"ID":"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff","Type":"ContainerStarted","Data":"eb017dcf95fb672dad30bb3df0103dc5b2b9d3132297bf0db5ec581715c61f50"}
Sep 09 20:53:02.600207 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:02.600176    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc" event={"ID":"b3434296-b3eb-48ef-b19f-95ce0c894107","Type":"ContainerStarted","Data":"fb7fad7837661e7c43a0ed6852bb8649f7373c33ddf713f496082ee28c24f8da"}
Sep 09 20:53:02.601766 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:02.601739    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h" event={"ID":"27d62ac1-f632-4428-adf5-25c91d1b8364","Type":"ContainerStarted","Data":"a7ffca038f4b0ddb9028ff7d92b88712a9bba04d59d5ffd2c12711f71c13fbb7"}
Sep 09 20:53:02.614543 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:02.614490    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254" podStartSLOduration=1.8274398729999999 podStartE2EDuration="4.614473073s" totalImagesPullingTime="2.7870332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:02.613707181 +0000 UTC m=+54.824250965" watchObservedRunningTime="2026-09-09 20:53:02.614473073 +0000 UTC m=+54.825016856"
Sep 09 20:53:02.631202 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:02.631139    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h" podStartSLOduration=2.762702147 podStartE2EDuration="5.631119188s" totalImagesPullingTime="2.868417041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:02.629565886 +0000 UTC m=+54.840109663" watchObservedRunningTime="2026-09-09 20:53:02.631119188 +0000 UTC m=+54.841662973"
Sep 09 20:53:02.645642 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:02.645577    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc" podStartSLOduration=2.809107348 podStartE2EDuration="5.645534374s" totalImagesPullingTime="2.836427026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:02.644350268 +0000 UTC m=+54.854894067" watchObservedRunningTime="2026-09-09 20:53:02.645534374 +0000 UTC m=+54.856078158"
Sep 09 20:53:03.257954 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.257916    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6hbfv"]
Sep 09 20:53:03.260749 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.260707    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.264331 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.264300    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:53:03.264592 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.264575    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99rg\""
Sep 09 20:53:03.264945 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.264923    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:53:03.282512 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.282481    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6hbfv"]
Sep 09 20:53:03.356502 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.356459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/376174b6-5720-4e8d-859a-9817ebad36b6-crio-socket\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.356695 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.356541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/376174b6-5720-4e8d-859a-9817ebad36b6-data-volume\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.356695 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.356623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.356695 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.356677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.356803 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.356730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqpq2\" (UniqueName: \"kubernetes.io/projected/376174b6-5720-4e8d-859a-9817ebad36b6-kube-api-access-wqpq2\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.457242 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.457203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.457242 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.457249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqpq2\" (UniqueName: \"kubernetes.io/projected/376174b6-5720-4e8d-859a-9817ebad36b6-kube-api-access-wqpq2\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.457496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.457299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/376174b6-5720-4e8d-859a-9817ebad36b6-crio-socket\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.457496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.457340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/376174b6-5720-4e8d-859a-9817ebad36b6-data-volume\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.457496 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.457371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.457496 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.457374    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:03.457496 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.457466    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.457717 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.457470    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:03.957450795 +0000 UTC m=+56.167994558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:03.457717 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.457531    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/376174b6-5720-4e8d-859a-9817ebad36b6-crio-socket\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.457717 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.457588    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:03.957539763 +0000 UTC m=+56.168083530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.457981 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.457717    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/376174b6-5720-4e8d-859a-9817ebad36b6-data-volume\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.474500 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.474473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqpq2\" (UniqueName: \"kubernetes.io/projected/376174b6-5720-4e8d-859a-9817ebad36b6-kube-api-access-wqpq2\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.962546 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.962501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.962753 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:03.962582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:03.962753 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.962654    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.962753 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.962705    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:03.962871 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.962754    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.962731461 +0000 UTC m=+57.173275223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:03.962871 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:03.962775    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.962765629 +0000 UTC m=+57.173309397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:04.971374 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:04.971327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:04.971814 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:04.971389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:04.971814 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:04.971454    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.971814 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:04.971521    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:04.971814 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:04.971523    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.971507547 +0000 UTC m=+59.182051309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.971814 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:04.971606    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:06.971591177 +0000 UTC m=+59.182134940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:05.229002 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:05.228929    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dq49d_796b5cc7-bfa3-4f0e-ba87-73d14a885180/dns-node-resolver/0.log"
Sep 09 20:53:05.778049 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:05.778006    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:53:05.778226 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:05.778162    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:53:05.778263 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:05.778234    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls podName:7b39b28e-07ba-4e0c-aa16-87d9b83d5a64 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:13.77821865 +0000 UTC m=+65.988762415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-6srvb" (UID: "7b39b28e-07ba-4e0c-aa16-87d9b83d5a64") : secret "samples-operator-tls" not found
Sep 09 20:53:06.429643 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:06.429614    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xtcgf_9cfac31b-51fd-4662-8938-4f217ab90c2d/node-ca/0.log"
Sep 09 20:53:06.988511 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:06.988467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:06.988746 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:06.988534    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:06.988746 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:06.988616    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:06.988746 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:06.988685    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.988669932 +0000 UTC m=+63.199213695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:06.988746 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:06.988686    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:06.988746 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:06.988735    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.988721456 +0000 UTC m=+63.199265217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:07.228371 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:07.228335    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-scgvd_c7177e33-d467-4ed6-bbf5-9687e57c09e4/serve-healthcheck-canary/0.log"
Sep 09 20:53:09.522097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:09.522066    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbk97"
Sep 09 20:53:11.022312 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:11.022273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:11.022803 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:11.022338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:11.022803 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:11.022393    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:11.022803 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:11.022456    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:19.022439626 +0000 UTC m=+71.232983407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:11.022803 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:11.022467    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:11.022803 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:11.022540    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:19.022519227 +0000 UTC m=+71.233062990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:13.037288 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.037240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:53:13.039835 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.039815    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:53:13.047757 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:13.047734    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:53:13.047824 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:13.047813    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs podName:6d4bd7ab-dd8f-45d6-a6bc-23642e40c927 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:17.047795853 +0000 UTC m=+129.258339631 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs") pod "network-metrics-daemon-xr4nz" (UID: "6d4bd7ab-dd8f-45d6-a6bc-23642e40c927") : secret "metrics-daemon-secret" not found
Sep 09 20:53:13.137910 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.137876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:53:13.138027 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.138009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:53:13.140756 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.140736    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:53:13.140851 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.140778    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:53:13.151163 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.151141    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cd6adce-d09e-42db-8a98-edca9cfb2c26-original-pull-secret\") pod \"global-pull-secret-syncer-c45vx\" (UID: \"1cd6adce-d09e-42db-8a98-edca9cfb2c26\") " pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:53:13.151586 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.151566    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:53:13.162247 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.162226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-298s7\" (UniqueName: \"kubernetes.io/projected/ae4d02a8-3af7-4d46-85e9-675222272c94-kube-api-access-298s7\") pod \"network-check-target-k2kqd\" (UID: \"ae4d02a8-3af7-4d46-85e9-675222272c94\") " pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:53:13.311253 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.311224    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7m4wj\""
Sep 09 20:53:13.314204 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.314183    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c45vx"
Sep 09 20:53:13.319342 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.319322    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:53:13.616578 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.616482    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c45vx"]
Sep 09 20:53:13.620199 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:53:13.620173    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd6adce_d09e_42db_8a98_edca9cfb2c26.slice/crio-8eb6295c65fd759bda57ee964821f6a4f21b0a1444b47067fa93920e40bac8b5 WatchSource:0}: Error finding container 8eb6295c65fd759bda57ee964821f6a4f21b0a1444b47067fa93920e40bac8b5: Status 404 returned error can't find the container with id 8eb6295c65fd759bda57ee964821f6a4f21b0a1444b47067fa93920e40bac8b5
Sep 09 20:53:13.630860 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.630825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c45vx" event={"ID":"1cd6adce-d09e-42db-8a98-edca9cfb2c26","Type":"ContainerStarted","Data":"8eb6295c65fd759bda57ee964821f6a4f21b0a1444b47067fa93920e40bac8b5"}
Sep 09 20:53:13.637599 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.637575    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k2kqd"]
Sep 09 20:53:13.640346 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:53:13.640319    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4d02a8_3af7_4d46_85e9_675222272c94.slice/crio-c88f924fcddb36aa4a07aa49da01bf08333625a7dcd9ae6f46676dee2cb170a6 WatchSource:0}: Error finding container c88f924fcddb36aa4a07aa49da01bf08333625a7dcd9ae6f46676dee2cb170a6: Status 404 returned error can't find the container with id c88f924fcddb36aa4a07aa49da01bf08333625a7dcd9ae6f46676dee2cb170a6
Sep 09 20:53:13.843854 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.843819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:53:13.846318 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.846291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b39b28e-07ba-4e0c-aa16-87d9b83d5a64-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-6srvb\" (UID: \"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:53:13.968452 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.968377    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-thljz\""
Sep 09 20:53:13.976608 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:13.976568    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"
Sep 09 20:53:14.146905 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.146871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:53:14.147379 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.146929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:53:14.149785 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.149756    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5-metrics-tls\") pod \"dns-default-7fppx\" (UID: \"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5\") " pod="openshift-dns/dns-default-7fppx"
Sep 09 20:53:14.150207 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.150188    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"image-registry-55bc7d5dcd-s4c89\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") " pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:53:14.251048 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.250957    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb"]
Sep 09 20:53:14.328196 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.328169    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lr6q6\""
Sep 09 20:53:14.333145 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.333121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:53:14.341455 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.341433    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vdcc\""
Sep 09 20:53:14.349186 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.349153    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7fppx"
Sep 09 20:53:14.623492 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.623449    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bc7d5dcd-s4c89"]
Sep 09 20:53:14.630405 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:53:14.630374    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7fab98_bbd0_48fd_aa24_384491b0a589.slice/crio-d752cb69829f5da9e76dda6f98f1062c096e063172f42f6ed2997d4679f1e393 WatchSource:0}: Error finding container d752cb69829f5da9e76dda6f98f1062c096e063172f42f6ed2997d4679f1e393: Status 404 returned error can't find the container with id d752cb69829f5da9e76dda6f98f1062c096e063172f42f6ed2997d4679f1e393
Sep 09 20:53:14.635696 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.635668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89" event={"ID":"1b7fab98-bbd0-48fd-aa24-384491b0a589","Type":"ContainerStarted","Data":"d752cb69829f5da9e76dda6f98f1062c096e063172f42f6ed2997d4679f1e393"}
Sep 09 20:53:14.637124 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.637067    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb" event={"ID":"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64","Type":"ContainerStarted","Data":"ca596dfbba047c0ab91d5dfbb52f96f3237d93f5bcf3acb1125b45559542bfc0"}
Sep 09 20:53:14.638630 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.638577    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k2kqd" event={"ID":"ae4d02a8-3af7-4d46-85e9-675222272c94","Type":"ContainerStarted","Data":"c88f924fcddb36aa4a07aa49da01bf08333625a7dcd9ae6f46676dee2cb170a6"}
Sep 09 20:53:14.644184 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:14.644125    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7fppx"]
Sep 09 20:53:14.648474 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:53:14.648437    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3fa8e7_c21c_43fa_b3e0_adad1ae499b5.slice/crio-c62900f3fab0821f8954efb836d786e771a7d70d28660453495308f596619f6e WatchSource:0}: Error finding container c62900f3fab0821f8954efb836d786e771a7d70d28660453495308f596619f6e: Status 404 returned error can't find the container with id c62900f3fab0821f8954efb836d786e771a7d70d28660453495308f596619f6e
Sep 09 20:53:15.645617 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:15.645577    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89" event={"ID":"1b7fab98-bbd0-48fd-aa24-384491b0a589","Type":"ContainerStarted","Data":"ec245a91eed31afce941a4ff4441723966e378e4b53e2f52df4aec41d5246d5c"}
Sep 09 20:53:15.646184 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:15.645873    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:53:15.649347 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:15.649314    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fppx" event={"ID":"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5","Type":"ContainerStarted","Data":"c62900f3fab0821f8954efb836d786e771a7d70d28660453495308f596619f6e"}
Sep 09 20:53:15.673416 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:15.673360    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89" podStartSLOduration=66.673343127 podStartE2EDuration="1m6.673343127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:15.672160929 +0000 UTC m=+67.882704737" watchObservedRunningTime="2026-09-09 20:53:15.673343127 +0000 UTC m=+67.883886912"
Sep 09 20:53:19.091540 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.091497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:19.092055 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.091604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:19.092055 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:19.091737    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:19.092055 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:19.091804    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls podName:376174b6-5720-4e8d-859a-9817ebad36b6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:35.091785878 +0000 UTC m=+87.302329661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6hbfv" (UID: "376174b6-5720-4e8d-859a-9817ebad36b6") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:19.092226 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.092142    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/376174b6-5720-4e8d-859a-9817ebad36b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:19.667548 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.667463    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb" event={"ID":"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64","Type":"ContainerStarted","Data":"f16f92386985f961211b4e3cd8a4fc12317801e925eb5a140df02a125cb49553"}
Sep 09 20:53:19.667548 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.667519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb" event={"ID":"7b39b28e-07ba-4e0c-aa16-87d9b83d5a64","Type":"ContainerStarted","Data":"5de7951deca2d65b25fdd7bffa7514013b7aa8acbc4df1d5c1e5b6a8f0869f48"}
Sep 09 20:53:19.669069 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.669045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k2kqd" event={"ID":"ae4d02a8-3af7-4d46-85e9-675222272c94","Type":"ContainerStarted","Data":"024fb537807d879545c84c1fcdbd16295559b1b124867cb182f02ce544f40434"}
Sep 09 20:53:19.669178 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.669101    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:53:19.670963 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.670941    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fppx" event={"ID":"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5","Type":"ContainerStarted","Data":"4b381b9785174a8f8a9602d6a1211ff2fb2aa883e1112662e162e9d357f608f9"}
Sep 09 20:53:19.671058 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.670982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7fppx" event={"ID":"dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5","Type":"ContainerStarted","Data":"43e68705542bb99e7f753a0d85678c5d2fdac11b60d3b3b77badf0d901a30288"}
Sep 09 20:53:19.671058 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.671030    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7fppx"
Sep 09 20:53:19.672430 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.672411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c45vx" event={"ID":"1cd6adce-d09e-42db-8a98-edca9cfb2c26","Type":"ContainerStarted","Data":"a38a901f1f0bf615abefdd0ca2ef084be0c236bf0806a3c0d94b8e7ca13aff49"}
Sep 09 20:53:19.687791 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.687738    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-6srvb" podStartSLOduration=17.687316605 podStartE2EDuration="22.687723769s" totalImagesPullingTime="5.000407164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:19.686575827 +0000 UTC m=+71.897119615" watchObservedRunningTime="2026-09-09 20:53:19.687723769 +0000 UTC m=+71.898267598"
Sep 09 20:53:19.746413 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.746356    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c45vx" podStartSLOduration=65.041669409 podStartE2EDuration="1m10.746337288s" totalImagesPullingTime="5.704667879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:19.714814466 +0000 UTC m=+71.925358251" watchObservedRunningTime="2026-09-09 20:53:19.746337288 +0000 UTC m=+71.956881073"
Sep 09 20:53:19.771418 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.771368    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k2kqd" podStartSLOduration=65.103319839 podStartE2EDuration="1m10.771354069s" totalImagesPullingTime="5.66803423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:19.746661127 +0000 UTC m=+71.957204913" watchObservedRunningTime="2026-09-09 20:53:19.771354069 +0000 UTC m=+71.981897852"
Sep 09 20:53:19.771996 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:19.771968    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7fppx" podStartSLOduration=33.110294311 podStartE2EDuration="37.771959474s" totalImagesPullingTime="4.661665163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:19.770624953 +0000 UTC m=+71.981168737" watchObservedRunningTime="2026-09-09 20:53:19.771959474 +0000 UTC m=+71.982503258"
Sep 09 20:53:26.677452 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.677418    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"]
Sep 09 20:53:26.682052 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.682036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.688396 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.688270    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:53:26.688699 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.688681    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:53:26.689287 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.689270    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:53:26.691250 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.691232    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:53:26.691781 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.691767    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:53:26.692474 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.692458    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:53:26.692744 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.692728    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:53:26.705708 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.705681    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"]
Sep 09 20:53:26.751346 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.751312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-hub\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.751528 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.751383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-ca\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.751528 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.751432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcnw\" (UniqueName: \"kubernetes.io/projected/30dd38b1-7328-43e9-9a20-c738f7056072-kube-api-access-hbcnw\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.751528 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.751493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.751741 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.751600    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.751741 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.751637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dd38b1-7328-43e9-9a20-c738f7056072-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.852537 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.852494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.852537 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.852534    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dd38b1-7328-43e9-9a20-c738f7056072-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.852820 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.852679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-hub\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.852820 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.852805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-ca\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.852923 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.852830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcnw\" (UniqueName: \"kubernetes.io/projected/30dd38b1-7328-43e9-9a20-c738f7056072-kube-api-access-hbcnw\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.852923 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.852857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.853296 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.853269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/30dd38b1-7328-43e9-9a20-c738f7056072-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.855536 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.855511    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-ca\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.855687 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.855664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.855746 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.855718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.855804 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.855760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/30dd38b1-7328-43e9-9a20-c738f7056072-hub\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.866391 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.866367    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcnw\" (UniqueName: \"kubernetes.io/projected/30dd38b1-7328-43e9-9a20-c738f7056072-kube-api-access-hbcnw\") pod \"cluster-proxy-proxy-agent-6cb455db9c-km9jr\" (UID: \"30dd38b1-7328-43e9-9a20-c738f7056072\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:26.996887 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:26.996789    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"
Sep 09 20:53:27.278517 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:27.278438    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr"]
Sep 09 20:53:27.282265 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:53:27.282238    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30dd38b1_7328_43e9_9a20_c738f7056072.slice/crio-b52eaaf6932152731497a3f0591f88dba63cc2bfe9c0db050718fd9780b9b461 WatchSource:0}: Error finding container b52eaaf6932152731497a3f0591f88dba63cc2bfe9c0db050718fd9780b9b461: Status 404 returned error can't find the container with id b52eaaf6932152731497a3f0591f88dba63cc2bfe9c0db050718fd9780b9b461
Sep 09 20:53:27.698900 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:27.698862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr" event={"ID":"30dd38b1-7328-43e9-9a20-c738f7056072","Type":"ContainerStarted","Data":"b52eaaf6932152731497a3f0591f88dba63cc2bfe9c0db050718fd9780b9b461"}
Sep 09 20:53:29.680764 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:29.678985    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7fppx"
Sep 09 20:53:31.714773 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:31.714737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr" event={"ID":"30dd38b1-7328-43e9-9a20-c738f7056072","Type":"ContainerStarted","Data":"cf77043e0ef84035d973a15ab8624bafd8fe3f8b24188dd10232ab469dfe9c26"}
Sep 09 20:53:31.715261 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:31.714779    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr" event={"ID":"30dd38b1-7328-43e9-9a20-c738f7056072","Type":"ContainerStarted","Data":"c45e7b83542da11fbb50e05d69e468ac278bc3c93b6038708d2ab7073759d3ee"}
Sep 09 20:53:31.715261 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:31.714794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr" event={"ID":"30dd38b1-7328-43e9-9a20-c738f7056072","Type":"ContainerStarted","Data":"e7c50f6ce069da86a4d1595b2652ba4ecd84bb92859cee5c36d0dcfcc273d13c"}
Sep 09 20:53:31.736699 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:31.736648    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cb455db9c-km9jr" podStartSLOduration=2.351838119 podStartE2EDuration="5.736632864s" totalImagesPullingTime="3.384794745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:31.73606246 +0000 UTC m=+83.946606245" watchObservedRunningTime="2026-09-09 20:53:31.736632864 +0000 UTC m=+83.947176647"
Sep 09 20:53:34.337989 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:34.337954    2573 patch_prober.go:28] interesting pod/image-registry-55bc7d5dcd-s4c89 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:53:34.338373 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:34.338010    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89" podUID="1b7fab98-bbd0-48fd-aa24-384491b0a589" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:53:35.118956 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:35.118917    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:35.121455 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:35.121426    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/376174b6-5720-4e8d-859a-9817ebad36b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6hbfv\" (UID: \"376174b6-5720-4e8d-859a-9817ebad36b6\") " pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:35.374793 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:35.374714    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s99rg\""
Sep 09 20:53:35.383156 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:35.383125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6hbfv"
Sep 09 20:53:35.697342 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:35.697125    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6hbfv"]
Sep 09 20:53:35.699894 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:53:35.699867    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376174b6_5720_4e8d_859a_9817ebad36b6.slice/crio-49d01cc31c5f72c93e4d1678eae4e9356cc84e6f3abb0ba8ddfd91680f079e3b WatchSource:0}: Error finding container 49d01cc31c5f72c93e4d1678eae4e9356cc84e6f3abb0ba8ddfd91680f079e3b: Status 404 returned error can't find the container with id 49d01cc31c5f72c93e4d1678eae4e9356cc84e6f3abb0ba8ddfd91680f079e3b
Sep 09 20:53:35.728332 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:35.728301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hbfv" event={"ID":"376174b6-5720-4e8d-859a-9817ebad36b6","Type":"ContainerStarted","Data":"49d01cc31c5f72c93e4d1678eae4e9356cc84e6f3abb0ba8ddfd91680f079e3b"}
Sep 09 20:53:36.660490 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:36.660462    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:53:36.734216 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:36.734185    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hbfv" event={"ID":"376174b6-5720-4e8d-859a-9817ebad36b6","Type":"ContainerStarted","Data":"6bb18325706a29d5207cbf522946d4e0abde2a8a92525b115e33aeeef900f2bf"}
Sep 09 20:53:37.740379 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:37.740338    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hbfv" event={"ID":"376174b6-5720-4e8d-859a-9817ebad36b6","Type":"ContainerStarted","Data":"1214f39f19dece7e50e1c49bb391964d42e0b522fe988e6f10fbc1117f7e7609"}
Sep 09 20:53:38.745518 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:38.745485    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6hbfv" event={"ID":"376174b6-5720-4e8d-859a-9817ebad36b6","Type":"ContainerStarted","Data":"715eb56d4d061573ccc17628a2395a5af8cbd9dd60bacf7df77b007b22381091"}
Sep 09 20:53:38.772287 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:38.772026    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6hbfv" podStartSLOduration=33.621185452 podStartE2EDuration="35.772005494s" totalImagesPullingTime="2.150820042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:38.770166702 +0000 UTC m=+90.980710480" watchObservedRunningTime="2026-09-09 20:53:38.772005494 +0000 UTC m=+90.982549279"
Sep 09 20:53:48.647235 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:48.647200    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bc7d5dcd-s4c89"]
Sep 09 20:53:50.679185 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:50.679152    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k2kqd"
Sep 09 20:53:53.956331 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.956300    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xvwmr"]
Sep 09 20:53:53.959700 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.959682    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:53.963205 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.963182    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:53:53.963658 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.963639    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:53:53.963767 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.963683    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v2wxl\""
Sep 09 20:53:53.964219 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.964202    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:53:53.964993 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.964965    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:53:53.965178 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.965163    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:53:53.965681 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:53.965662    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:53:54.062863 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.062831    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.062863 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.062870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-tls\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.063091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.062888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-wtmp\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.063091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.062908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-sys\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.063091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.062929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-root\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.063091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.062978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-metrics-client-ca\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.063091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.063001    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n86q\" (UniqueName: \"kubernetes.io/projected/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-kube-api-access-5n86q\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.063091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.063047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-textfile\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.063091 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.063084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.163904 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.163863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n86q\" (UniqueName: \"kubernetes.io/projected/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-kube-api-access-5n86q\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.163912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-textfile\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.163955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.163977    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-tls\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164026    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-wtmp\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164052    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-sys\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164097 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-root\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164367 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-metrics-client-ca\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164367 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:54.164123    2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:54.164367 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164180    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-sys\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164367 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164167    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-root\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164367 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-wtmp\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164367 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:53:54.164246    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-tls podName:c9ee83c4-5d84-4f37-80bb-2d28a97288d4 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:54.664209375 +0000 UTC m=+106.874753154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-tls") pod "node-exporter-xvwmr" (UID: "c9ee83c4-5d84-4f37-80bb-2d28a97288d4") : secret "node-exporter-tls" not found
Sep 09 20:53:54.164636 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164377    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-textfile\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164693 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164670    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-metrics-client-ca\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.164746 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.164726    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.166640 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.166621    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.223003 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.222924    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n86q\" (UniqueName: \"kubernetes.io/projected/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-kube-api-access-5n86q\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.668663 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.668616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-tls\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.670989 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.670968    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c9ee83c4-5d84-4f37-80bb-2d28a97288d4-node-exporter-tls\") pod \"node-exporter-xvwmr\" (UID: \"c9ee83c4-5d84-4f37-80bb-2d28a97288d4\") " pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.868854 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:54.868823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xvwmr"
Sep 09 20:53:54.878699 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:53:54.878666    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ee83c4_5d84_4f37_80bb_2d28a97288d4.slice/crio-dd76c1688d0986c9b666d69722302e7b92c8aa631c1b09889d1f1437aab9294b WatchSource:0}: Error finding container dd76c1688d0986c9b666d69722302e7b92c8aa631c1b09889d1f1437aab9294b: Status 404 returned error can't find the container with id dd76c1688d0986c9b666d69722302e7b92c8aa631c1b09889d1f1437aab9294b
Sep 09 20:53:55.804099 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:55.804057    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvwmr" event={"ID":"c9ee83c4-5d84-4f37-80bb-2d28a97288d4","Type":"ContainerStarted","Data":"dd76c1688d0986c9b666d69722302e7b92c8aa631c1b09889d1f1437aab9294b"}
Sep 09 20:53:56.808804 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:56.808767    2573 generic.go:356] "Generic (PLEG): container finished" podID="c9ee83c4-5d84-4f37-80bb-2d28a97288d4" containerID="408540a5fab4b792d5237b1d3ebd81cc02bfec9b938c12a9e48788126861dd37" exitCode=0
Sep 09 20:53:56.809201 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:56.808862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvwmr" event={"ID":"c9ee83c4-5d84-4f37-80bb-2d28a97288d4","Type":"ContainerDied","Data":"408540a5fab4b792d5237b1d3ebd81cc02bfec9b938c12a9e48788126861dd37"}
Sep 09 20:53:57.813983 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:57.813948    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvwmr" event={"ID":"c9ee83c4-5d84-4f37-80bb-2d28a97288d4","Type":"ContainerStarted","Data":"0182160fba4ce9a09a55ecbfac410687102aface32ca22f22be2df3f4f45c82d"}
Sep 09 20:53:57.814411 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:57.813998    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvwmr" event={"ID":"c9ee83c4-5d84-4f37-80bb-2d28a97288d4","Type":"ContainerStarted","Data":"c26acb686149c52741e2f8f03a18c0558d8ebb5e3ab08dceb2a908185521dfdc"}
Sep 09 20:53:57.847103 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:53:57.847018    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xvwmr" podStartSLOduration=3.3447099590000002 podStartE2EDuration="4.847001805s" totalImagesPullingTime="1.430601289s" totalInitContainerRuntime="71.690557ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:57.845602117 +0000 UTC m=+110.056145902" watchObservedRunningTime="2026-09-09 20:53:57.847001805 +0000 UTC m=+110.057545589"
Sep 09 20:54:08.851634 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:08.851588    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3434296-b3eb-48ef-b19f-95ce0c894107" containerID="fb7fad7837661e7c43a0ed6852bb8649f7373c33ddf713f496082ee28c24f8da" exitCode=0
Sep 09 20:54:08.852145 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:08.851650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc" event={"ID":"b3434296-b3eb-48ef-b19f-95ce0c894107","Type":"ContainerDied","Data":"fb7fad7837661e7c43a0ed6852bb8649f7373c33ddf713f496082ee28c24f8da"}
Sep 09 20:54:08.852145 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:08.852024    2573 scope.go:117] "RemoveContainer" containerID="fb7fad7837661e7c43a0ed6852bb8649f7373c33ddf713f496082ee28c24f8da"
Sep 09 20:54:09.857114 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:09.857082    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-dgrqc" event={"ID":"b3434296-b3eb-48ef-b19f-95ce0c894107","Type":"ContainerStarted","Data":"b1dfb38295c728c56e94d15990594c03c8a17e1614f3df4c5cd3dc820ea0ef10"}
Sep 09 20:54:13.667473 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:13.667406    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89" podUID="1b7fab98-bbd0-48fd-aa24-384491b0a589" containerName="registry" containerID="cri-o://ec245a91eed31afce941a4ff4441723966e378e4b53e2f52df4aec41d5246d5c" gracePeriod=30
Sep 09 20:54:13.871542 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:13.871506    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b7fab98-bbd0-48fd-aa24-384491b0a589" containerID="ec245a91eed31afce941a4ff4441723966e378e4b53e2f52df4aec41d5246d5c" exitCode=0
Sep 09 20:54:13.871542 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:13.871529    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89" event={"ID":"1b7fab98-bbd0-48fd-aa24-384491b0a589","Type":"ContainerDied","Data":"ec245a91eed31afce941a4ff4441723966e378e4b53e2f52df4aec41d5246d5c"}
Sep 09 20:54:13.913399 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:13.913373    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:54:14.008370 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008279    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg7s7\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-kube-api-access-tg7s7\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008370 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008324    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008370 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008344    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-trusted-ca\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008370 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008372    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b7fab98-bbd0-48fd-aa24-384491b0a589-ca-trust-extracted\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008709 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008409    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-image-registry-private-configuration\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008709 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008442    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-certificates\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008709 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008466    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-installation-pull-secrets\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008709 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008510    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-bound-sa-token\") pod \"1b7fab98-bbd0-48fd-aa24-384491b0a589\" (UID: \"1b7fab98-bbd0-48fd-aa24-384491b0a589\") "
Sep 09 20:54:14.008964 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008939    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:14.009023 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.008967    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:14.011163 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.011130    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:14.011292 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.011186    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:14.011292 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.011232    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:14.011443 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.011418    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-kube-api-access-tg7s7" (OuterVolumeSpecName: "kube-api-access-tg7s7") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "kube-api-access-tg7s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:14.011543 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.011464    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:14.017052 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.017024    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7fab98-bbd0-48fd-aa24-384491b0a589-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1b7fab98-bbd0-48fd-aa24-384491b0a589" (UID: "1b7fab98-bbd0-48fd-aa24-384491b0a589"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:14.109936 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109899    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7s7\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-kube-api-access-tg7s7\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.109936 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109929    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.109936 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109938    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-trusted-ca\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.110155 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109946    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1b7fab98-bbd0-48fd-aa24-384491b0a589-ca-trust-extracted\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.110155 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109956    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-image-registry-private-configuration\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.110155 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109965    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1b7fab98-bbd0-48fd-aa24-384491b0a589-registry-certificates\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.110155 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109973    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1b7fab98-bbd0-48fd-aa24-384491b0a589-installation-pull-secrets\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.110155 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.109981    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b7fab98-bbd0-48fd-aa24-384491b0a589-bound-sa-token\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:54:14.876659 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.876633    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89"
Sep 09 20:54:14.877086 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.876625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc7d5dcd-s4c89" event={"ID":"1b7fab98-bbd0-48fd-aa24-384491b0a589","Type":"ContainerDied","Data":"d752cb69829f5da9e76dda6f98f1062c096e063172f42f6ed2997d4679f1e393"}
Sep 09 20:54:14.877086 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.876766    2573 scope.go:117] "RemoveContainer" containerID="ec245a91eed31afce941a4ff4441723966e378e4b53e2f52df4aec41d5246d5c"
Sep 09 20:54:14.891738 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.891713    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bc7d5dcd-s4c89"]
Sep 09 20:54:14.896000 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:14.895978    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55bc7d5dcd-s4c89"]
Sep 09 20:54:16.286892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:16.286856    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7fab98-bbd0-48fd-aa24-384491b0a589" path="/var/lib/kubelet/pods/1b7fab98-bbd0-48fd-aa24-384491b0a589/volumes"
Sep 09 20:54:17.129457 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.129416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:54:17.131829 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.131801    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d4bd7ab-dd8f-45d6-a6bc-23642e40c927-metrics-certs\") pod \"network-metrics-daemon-xr4nz\" (UID: \"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927\") " pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:54:17.222134 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.222098    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w9kf4\""
Sep 09 20:54:17.229275 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.229252    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xr4nz"
Sep 09 20:54:17.503153 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.503074    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xr4nz"]
Sep 09 20:54:17.506049 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:54:17.506013    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4bd7ab_dd8f_45d6_a6bc_23642e40c927.slice/crio-a7e5bd2ed5b09fd7553ed4a93f530c9ab8b4c07bb4156723c60c9d0b26423cb8 WatchSource:0}: Error finding container a7e5bd2ed5b09fd7553ed4a93f530c9ab8b4c07bb4156723c60c9d0b26423cb8: Status 404 returned error can't find the container with id a7e5bd2ed5b09fd7553ed4a93f530c9ab8b4c07bb4156723c60c9d0b26423cb8
Sep 09 20:54:17.891579 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.891519    2573 generic.go:356] "Generic (PLEG): container finished" podID="b1e0d315-b0c6-4e02-9684-b75d86e5b0ff" containerID="eb017dcf95fb672dad30bb3df0103dc5b2b9d3132297bf0db5ec581715c61f50" exitCode=0
Sep 09 20:54:17.891831 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.891617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254" event={"ID":"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff","Type":"ContainerDied","Data":"eb017dcf95fb672dad30bb3df0103dc5b2b9d3132297bf0db5ec581715c61f50"}
Sep 09 20:54:17.892354 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.892005    2573 scope.go:117] "RemoveContainer" containerID="eb017dcf95fb672dad30bb3df0103dc5b2b9d3132297bf0db5ec581715c61f50"
Sep 09 20:54:17.893338 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:17.893306    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xr4nz" event={"ID":"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927","Type":"ContainerStarted","Data":"a7e5bd2ed5b09fd7553ed4a93f530c9ab8b4c07bb4156723c60c9d0b26423cb8"}
Sep 09 20:54:18.898544 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:18.898507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xr4nz" event={"ID":"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927","Type":"ContainerStarted","Data":"b75c8ed830271b6f20a272c45192c8a0c0042595d0bc92ca244b286f3b2a5ba5"}
Sep 09 20:54:18.899018 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:18.898587    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xr4nz" event={"ID":"6d4bd7ab-dd8f-45d6-a6bc-23642e40c927","Type":"ContainerStarted","Data":"c23013eb2022257a650ee5edcafac70c5f1c8f09b2ad0e64cf2a2bda1349a648"}
Sep 09 20:54:18.900189 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:18.900170    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-z4254" event={"ID":"b1e0d315-b0c6-4e02-9684-b75d86e5b0ff","Type":"ContainerStarted","Data":"18a06904a5e5e15ad5ffdac1b3c4a7ef9640a4685f68aa930542ce8ccffeadea"}
Sep 09 20:54:18.938429 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:18.938371    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xr4nz" podStartSLOduration=130.048191657 podStartE2EDuration="2m10.938350472s" totalImagesPullingTime="890.158815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:18.917312997 +0000 UTC m=+131.127856802" watchObservedRunningTime="2026-09-09 20:54:18.938350472 +0000 UTC m=+131.148894257"
Sep 09 20:54:27.930895 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:27.930858    2573 generic.go:356] "Generic (PLEG): container finished" podID="27d62ac1-f632-4428-adf5-25c91d1b8364" containerID="a7ffca038f4b0ddb9028ff7d92b88712a9bba04d59d5ffd2c12711f71c13fbb7" exitCode=0
Sep 09 20:54:27.931366 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:27.930908    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h" event={"ID":"27d62ac1-f632-4428-adf5-25c91d1b8364","Type":"ContainerDied","Data":"a7ffca038f4b0ddb9028ff7d92b88712a9bba04d59d5ffd2c12711f71c13fbb7"}
Sep 09 20:54:27.931366 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:27.931279    2573 scope.go:117] "RemoveContainer" containerID="a7ffca038f4b0ddb9028ff7d92b88712a9bba04d59d5ffd2c12711f71c13fbb7"
Sep 09 20:54:28.936379 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:54:28.936347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-2qt7h" event={"ID":"27d62ac1-f632-4428-adf5-25c91d1b8364","Type":"ContainerStarted","Data":"d0c6e109c540876eb6bc9963592d6cc83d5d4c255564683de9e92e330700e148"}
Sep 09 20:57:08.222654 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:08.222627    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 20:57:08.223107 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:08.222627    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 20:57:08.229666 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:08.229645    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:57:50.462351 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.462317    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"]
Sep 09 20:57:50.462792 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.462568    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7fab98-bbd0-48fd-aa24-384491b0a589" containerName="registry"
Sep 09 20:57:50.462792 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.462579    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7fab98-bbd0-48fd-aa24-384491b0a589" containerName="registry"
Sep 09 20:57:50.462792 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.462628    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7fab98-bbd0-48fd-aa24-384491b0a589" containerName="registry"
Sep 09 20:57:50.465343 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.465328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.470516 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.470491    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:57:50.470685 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.470491    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:57:50.470685 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.470546    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8kvrg\""
Sep 09 20:57:50.481965 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.481934    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"]
Sep 09 20:57:50.544433 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.544398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.544627 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.544454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4tz\" (UniqueName: \"kubernetes.io/projected/ed88f4af-7cb0-46c3-ba60-afc310db0e55-kube-api-access-9d4tz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.544627 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.544524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.645394 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.645360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.645582 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.645404    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.645582 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.645436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4tz\" (UniqueName: \"kubernetes.io/projected/ed88f4af-7cb0-46c3-ba60-afc310db0e55-kube-api-access-9d4tz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.645795 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.645774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.645837 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.645792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.663431 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.663409    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4tz\" (UniqueName: \"kubernetes.io/projected/ed88f4af-7cb0-46c3-ba60-afc310db0e55-kube-api-access-9d4tz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:50.774061 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:50.773973    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:57:51.067547 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:51.067519    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"]
Sep 09 20:57:51.070814 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:57:51.070788    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded88f4af_7cb0_46c3_ba60_afc310db0e55.slice/crio-6e6e356638650cab3fdf1868af1144609b8969e752600e4d3c18aeb9791eed1a WatchSource:0}: Error finding container 6e6e356638650cab3fdf1868af1144609b8969e752600e4d3c18aeb9791eed1a: Status 404 returned error can't find the container with id 6e6e356638650cab3fdf1868af1144609b8969e752600e4d3c18aeb9791eed1a
Sep 09 20:57:51.072788 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:51.072769    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:57:51.638039 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:51.637999    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m" event={"ID":"ed88f4af-7cb0-46c3-ba60-afc310db0e55","Type":"ContainerStarted","Data":"6e6e356638650cab3fdf1868af1144609b8969e752600e4d3c18aeb9791eed1a"}
Sep 09 20:57:57.663233 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:57.663206    2573 generic.go:356] "Generic (PLEG): container finished" podID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerID="ac694a9955b4f02fbb9933428024d7d989b8aa19a73063adbb57bb6e5ad49ba7" exitCode=0
Sep 09 20:57:57.663583 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:57:57.663283    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m" event={"ID":"ed88f4af-7cb0-46c3-ba60-afc310db0e55","Type":"ContainerDied","Data":"ac694a9955b4f02fbb9933428024d7d989b8aa19a73063adbb57bb6e5ad49ba7"}
Sep 09 20:58:00.675225 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:00.675187    2573 generic.go:356] "Generic (PLEG): container finished" podID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerID="2a83469e81d5af4ad45a8babef081fe6ffd7fcb627c055267fbcbbc70c2f0acc" exitCode=0
Sep 09 20:58:00.675683 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:00.675267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m" event={"ID":"ed88f4af-7cb0-46c3-ba60-afc310db0e55","Type":"ContainerDied","Data":"2a83469e81d5af4ad45a8babef081fe6ffd7fcb627c055267fbcbbc70c2f0acc"}
Sep 09 20:58:09.718323 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:09.718240    2573 generic.go:356] "Generic (PLEG): container finished" podID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerID="77fc550caa9b436e0ee5c46c973f75385b59469071485fc6b2b1607e2c1a0642" exitCode=0
Sep 09 20:58:09.718323 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:09.718295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m" event={"ID":"ed88f4af-7cb0-46c3-ba60-afc310db0e55","Type":"ContainerDied","Data":"77fc550caa9b436e0ee5c46c973f75385b59469071485fc6b2b1607e2c1a0642"}
Sep 09 20:58:10.843095 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:10.843073    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:58:11.008779 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.008673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-util\") pod \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") "
Sep 09 20:58:11.008779 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.008756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d4tz\" (UniqueName: \"kubernetes.io/projected/ed88f4af-7cb0-46c3-ba60-afc310db0e55-kube-api-access-9d4tz\") pod \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") "
Sep 09 20:58:11.008967 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.008833    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-bundle\") pod \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\" (UID: \"ed88f4af-7cb0-46c3-ba60-afc310db0e55\") "
Sep 09 20:58:11.009445 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.009418    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-bundle" (OuterVolumeSpecName: "bundle") pod "ed88f4af-7cb0-46c3-ba60-afc310db0e55" (UID: "ed88f4af-7cb0-46c3-ba60-afc310db0e55"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:11.011042 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.011013    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed88f4af-7cb0-46c3-ba60-afc310db0e55-kube-api-access-9d4tz" (OuterVolumeSpecName: "kube-api-access-9d4tz") pod "ed88f4af-7cb0-46c3-ba60-afc310db0e55" (UID: "ed88f4af-7cb0-46c3-ba60-afc310db0e55"). InnerVolumeSpecName "kube-api-access-9d4tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:11.013720 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.013695    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-util" (OuterVolumeSpecName: "util") pod "ed88f4af-7cb0-46c3-ba60-afc310db0e55" (UID: "ed88f4af-7cb0-46c3-ba60-afc310db0e55"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:11.109772 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.109738    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-util\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:58:11.109772 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.109764    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4tz\" (UniqueName: \"kubernetes.io/projected/ed88f4af-7cb0-46c3-ba60-afc310db0e55-kube-api-access-9d4tz\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:58:11.109772 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.109775    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed88f4af-7cb0-46c3-ba60-afc310db0e55-bundle\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 20:58:11.727896 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.727870    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m"
Sep 09 20:58:11.728065 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.727865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1abmp4m" event={"ID":"ed88f4af-7cb0-46c3-ba60-afc310db0e55","Type":"ContainerDied","Data":"6e6e356638650cab3fdf1868af1144609b8969e752600e4d3c18aeb9791eed1a"}
Sep 09 20:58:11.728065 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:11.727996    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e6e356638650cab3fdf1868af1144609b8969e752600e4d3c18aeb9791eed1a"
Sep 09 20:58:21.624462 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624426    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-tw7vm"]
Sep 09 20:58:21.624892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624717    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerName="extract"
Sep 09 20:58:21.624892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624729    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerName="extract"
Sep 09 20:58:21.624892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624744    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerName="util"
Sep 09 20:58:21.624892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624750    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerName="util"
Sep 09 20:58:21.624892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624757    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerName="pull"
Sep 09 20:58:21.624892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624765    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerName="pull"
Sep 09 20:58:21.624892 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.624801    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed88f4af-7cb0-46c3-ba60-afc310db0e55" containerName="extract"
Sep 09 20:58:21.628195 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.628176    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.634133 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.634038    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:58:21.634133 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.634049    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:58:21.634133 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.634067    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:58:21.634923 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.634901    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:58:21.635015 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.634928    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-phrh6\""
Sep 09 20:58:21.635015 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.634950    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:58:21.639235 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.639214    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-tw7vm"]
Sep 09 20:58:21.785929 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.785895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvzl\" (UniqueName: \"kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-kube-api-access-drvzl\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.786105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.785945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2d341180-9b81-4bc6-b44c-e97c179a5d13-cabundle0\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.786105 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.786008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-certificates\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.887287 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.887196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-certificates\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.887287 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.887252    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvzl\" (UniqueName: \"kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-kube-api-access-drvzl\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.887287 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.887276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2d341180-9b81-4bc6-b44c-e97c179a5d13-cabundle0\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.887607 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:58:21.887369    2573 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:58:21.887607 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:58:21.887398    2573 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:58:21.887607 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:58:21.887410    2573 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-tw7vm: references non-existent secret key: ca.crt
Sep 09 20:58:21.887607 ip-10-0-129-205 kubenswrapper[2573]: E0909 20:58:21.887486    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-certificates podName:2d341180-9b81-4bc6-b44c-e97c179a5d13 nodeName:}" failed. No retries permitted until 2026-09-09 20:58:22.387466655 +0000 UTC m=+374.598010435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-certificates") pod "keda-operator-579c8958f7-tw7vm" (UID: "2d341180-9b81-4bc6-b44c-e97c179a5d13") : references non-existent secret key: ca.crt
Sep 09 20:58:21.887946 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.887929    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2d341180-9b81-4bc6-b44c-e97c179a5d13-cabundle0\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:21.904238 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:21.904215    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvzl\" (UniqueName: \"kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-kube-api-access-drvzl\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:22.392226 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.392190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-certificates\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:22.393613 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.393587    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"]
Sep 09 20:58:22.394978 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.394954    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2d341180-9b81-4bc6-b44c-e97c179a5d13-certificates\") pod \"keda-operator-579c8958f7-tw7vm\" (UID: \"2d341180-9b81-4bc6-b44c-e97c179a5d13\") " pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:22.395660 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.395646    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.398102 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.398084    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:58:22.427231 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.427200    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"]
Sep 09 20:58:22.492745 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.492699    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdswh\" (UniqueName: \"kubernetes.io/projected/ae27b951-cbff-444b-86c8-8f315b80b59a-kube-api-access-qdswh\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.492932 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.492786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ae27b951-cbff-444b-86c8-8f315b80b59a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.492932 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.492836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae27b951-cbff-444b-86c8-8f315b80b59a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.537354 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.537322    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:22.594189 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.594156    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae27b951-cbff-444b-86c8-8f315b80b59a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.594323 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.594214    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdswh\" (UniqueName: \"kubernetes.io/projected/ae27b951-cbff-444b-86c8-8f315b80b59a-kube-api-access-qdswh\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.594523 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.594457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ae27b951-cbff-444b-86c8-8f315b80b59a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.594942 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.594894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ae27b951-cbff-444b-86c8-8f315b80b59a-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.597959 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.597911    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae27b951-cbff-444b-86c8-8f315b80b59a-certificates\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.631058 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.631030    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdswh\" (UniqueName: \"kubernetes.io/projected/ae27b951-cbff-444b-86c8-8f315b80b59a-kube-api-access-qdswh\") pod \"keda-metrics-apiserver-77857b7b98-swzpf\" (UID: \"ae27b951-cbff-444b-86c8-8f315b80b59a\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.702647 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.702568    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-cxqz4"]
Sep 09 20:58:22.705217 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.705200    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:22.708925 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.708895    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:22.729535 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.729511    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:58:22.796956 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.796918    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/78feab2c-832a-4b19-8791-771e03f1535f-certificates\") pod \"keda-admission-6c4d864b86-cxqz4\" (UID: \"78feab2c-832a-4b19-8791-771e03f1535f\") " pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:22.797104 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.796984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6g5l\" (UniqueName: \"kubernetes.io/projected/78feab2c-832a-4b19-8791-771e03f1535f-kube-api-access-p6g5l\") pod \"keda-admission-6c4d864b86-cxqz4\" (UID: \"78feab2c-832a-4b19-8791-771e03f1535f\") " pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:22.853759 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.853715    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-cxqz4"]
Sep 09 20:58:22.898108 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.898075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6g5l\" (UniqueName: \"kubernetes.io/projected/78feab2c-832a-4b19-8791-771e03f1535f-kube-api-access-p6g5l\") pod \"keda-admission-6c4d864b86-cxqz4\" (UID: \"78feab2c-832a-4b19-8791-771e03f1535f\") " pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:22.898277 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.898140    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/78feab2c-832a-4b19-8791-771e03f1535f-certificates\") pod \"keda-admission-6c4d864b86-cxqz4\" (UID: \"78feab2c-832a-4b19-8791-771e03f1535f\") " pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:22.900760 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.900742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/78feab2c-832a-4b19-8791-771e03f1535f-certificates\") pod \"keda-admission-6c4d864b86-cxqz4\" (UID: \"78feab2c-832a-4b19-8791-771e03f1535f\") " pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:22.914093 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.914070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6g5l\" (UniqueName: \"kubernetes.io/projected/78feab2c-832a-4b19-8791-771e03f1535f-kube-api-access-p6g5l\") pod \"keda-admission-6c4d864b86-cxqz4\" (UID: \"78feab2c-832a-4b19-8791-771e03f1535f\") " pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:22.956764 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:22.956686    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-tw7vm"]
Sep 09 20:58:22.962959 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:58:22.962933    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d341180_9b81_4bc6_b44c_e97c179a5d13.slice/crio-ad1cb36a9cb37e06e7c3d1f802a5e38c5cc8704ec8f0bb35db3b38787594af45 WatchSource:0}: Error finding container ad1cb36a9cb37e06e7c3d1f802a5e38c5cc8704ec8f0bb35db3b38787594af45: Status 404 returned error can't find the container with id ad1cb36a9cb37e06e7c3d1f802a5e38c5cc8704ec8f0bb35db3b38787594af45
Sep 09 20:58:23.015574 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:23.015510    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:23.047746 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:58:23.047713    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae27b951_cbff_444b_86c8_8f315b80b59a.slice/crio-005f3ad773dfdb6414390a777d30fe5478f56606c3e177741352e219f3ad27ef WatchSource:0}: Error finding container 005f3ad773dfdb6414390a777d30fe5478f56606c3e177741352e219f3ad27ef: Status 404 returned error can't find the container with id 005f3ad773dfdb6414390a777d30fe5478f56606c3e177741352e219f3ad27ef
Sep 09 20:58:23.050372 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:23.050347    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"]
Sep 09 20:58:23.454731 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:23.454705    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-cxqz4"]
Sep 09 20:58:23.456288 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:58:23.456262    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78feab2c_832a_4b19_8791_771e03f1535f.slice/crio-89bcfc89a62b3f09c5b6b2058aed16180c1253448e61bd31c4296d0f832c87a0 WatchSource:0}: Error finding container 89bcfc89a62b3f09c5b6b2058aed16180c1253448e61bd31c4296d0f832c87a0: Status 404 returned error can't find the container with id 89bcfc89a62b3f09c5b6b2058aed16180c1253448e61bd31c4296d0f832c87a0
Sep 09 20:58:23.772165 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:23.772062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf" event={"ID":"ae27b951-cbff-444b-86c8-8f315b80b59a","Type":"ContainerStarted","Data":"005f3ad773dfdb6414390a777d30fe5478f56606c3e177741352e219f3ad27ef"}
Sep 09 20:58:23.773189 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:23.773167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-cxqz4" event={"ID":"78feab2c-832a-4b19-8791-771e03f1535f","Type":"ContainerStarted","Data":"89bcfc89a62b3f09c5b6b2058aed16180c1253448e61bd31c4296d0f832c87a0"}
Sep 09 20:58:23.774267 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:23.774246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-tw7vm" event={"ID":"2d341180-9b81-4bc6-b44c-e97c179a5d13","Type":"ContainerStarted","Data":"ad1cb36a9cb37e06e7c3d1f802a5e38c5cc8704ec8f0bb35db3b38787594af45"}
Sep 09 20:58:27.795054 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.795002    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf" event={"ID":"ae27b951-cbff-444b-86c8-8f315b80b59a","Type":"ContainerStarted","Data":"e3110e5a7dcae53009dc8671691c96862ba5debb66cba0d8eea3f9ed02d801e7"}
Sep 09 20:58:27.795524 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.795148    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:27.796779 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.796755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-cxqz4" event={"ID":"78feab2c-832a-4b19-8791-771e03f1535f","Type":"ContainerStarted","Data":"d37bdb6153e65f92cd65280cb97588af3cf6928550171d61c9eacf09e2ba8a48"}
Sep 09 20:58:27.796899 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.796867    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:58:27.798570 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.798524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-tw7vm" event={"ID":"2d341180-9b81-4bc6-b44c-e97c179a5d13","Type":"ContainerStarted","Data":"38af50573ab856ce8026355e4be5d3d72aa0573b32d8c458d080e747d69dc676"}
Sep 09 20:58:27.798688 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.798599    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:27.817935 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.817884    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf" podStartSLOduration=1.761019535 podStartE2EDuration="5.817868495s" totalImagesPullingTime="4.05684896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:27.816479394 +0000 UTC m=+380.027023178" watchObservedRunningTime="2026-09-09 20:58:27.817868495 +0000 UTC m=+380.028412279"
Sep 09 20:58:27.837072 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.837024    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-tw7vm" podStartSLOduration=2.690331589 podStartE2EDuration="6.837009556s" totalImagesPullingTime="4.146677967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:27.834997006 +0000 UTC m=+380.045540804" watchObservedRunningTime="2026-09-09 20:58:27.837009556 +0000 UTC m=+380.047553339"
Sep 09 20:58:27.858163 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:27.858114    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-cxqz4" podStartSLOduration=2.25557198 podStartE2EDuration="5.858098004s" totalImagesPullingTime="3.602526024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:27.856968393 +0000 UTC m=+380.067512178" watchObservedRunningTime="2026-09-09 20:58:27.858098004 +0000 UTC m=+380.068641788"
Sep 09 20:58:38.807480 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:38.807452    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-swzpf"
Sep 09 20:58:48.804698 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:48.804667    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-tw7vm"
Sep 09 20:58:48.805125 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:58:48.804735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-cxqz4"
Sep 09 20:59:45.980675 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:45.980633    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"]
Sep 09 20:59:45.982846 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:45.982824    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:45.984928 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:45.984908    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:59:45.985430 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:45.985412    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-4k7js\""
Sep 09 20:59:45.985796 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:45.985779    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:59:45.989662 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:45.989640    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:59:45.999617 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:45.999576    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"]
Sep 09 20:59:46.017460 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.017430    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/759f0e7b-4a99-4da5-934d-dc114a273e9c-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-8tfg9\" (UID: \"759f0e7b-4a99-4da5-934d-dc114a273e9c\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:46.017642 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.017475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvb24\" (UniqueName: \"kubernetes.io/projected/759f0e7b-4a99-4da5-934d-dc114a273e9c-kube-api-access-lvb24\") pod \"llmisvc-controller-manager-645f9c8fd6-8tfg9\" (UID: \"759f0e7b-4a99-4da5-934d-dc114a273e9c\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:46.118471 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.118436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/759f0e7b-4a99-4da5-934d-dc114a273e9c-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-8tfg9\" (UID: \"759f0e7b-4a99-4da5-934d-dc114a273e9c\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:46.118471 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.118473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvb24\" (UniqueName: \"kubernetes.io/projected/759f0e7b-4a99-4da5-934d-dc114a273e9c-kube-api-access-lvb24\") pod \"llmisvc-controller-manager-645f9c8fd6-8tfg9\" (UID: \"759f0e7b-4a99-4da5-934d-dc114a273e9c\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:46.121017 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.120993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/759f0e7b-4a99-4da5-934d-dc114a273e9c-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-8tfg9\" (UID: \"759f0e7b-4a99-4da5-934d-dc114a273e9c\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:46.127303 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.127282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvb24\" (UniqueName: \"kubernetes.io/projected/759f0e7b-4a99-4da5-934d-dc114a273e9c-kube-api-access-lvb24\") pod \"llmisvc-controller-manager-645f9c8fd6-8tfg9\" (UID: \"759f0e7b-4a99-4da5-934d-dc114a273e9c\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:46.294377 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.294351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:46.568385 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:46.568355    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"]
Sep 09 20:59:46.569316 ip-10-0-129-205 kubenswrapper[2573]: W0909 20:59:46.569289    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod759f0e7b_4a99_4da5_934d_dc114a273e9c.slice/crio-adfeff2e48b3e999320585c0d98faeb2e890cc8e6dda8d802c79678c2cf56b07 WatchSource:0}: Error finding container adfeff2e48b3e999320585c0d98faeb2e890cc8e6dda8d802c79678c2cf56b07: Status 404 returned error can't find the container with id adfeff2e48b3e999320585c0d98faeb2e890cc8e6dda8d802c79678c2cf56b07
Sep 09 20:59:47.092948 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:47.092909    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9" event={"ID":"759f0e7b-4a99-4da5-934d-dc114a273e9c","Type":"ContainerStarted","Data":"adfeff2e48b3e999320585c0d98faeb2e890cc8e6dda8d802c79678c2cf56b07"}
Sep 09 20:59:50.104934 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:50.104903    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9" event={"ID":"759f0e7b-4a99-4da5-934d-dc114a273e9c","Type":"ContainerStarted","Data":"32c4d3c7bb0366792f144162107d43abd802d238ec00414cf23ecaf8462b8757"}
Sep 09 20:59:50.105307 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:50.105076    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 20:59:50.123791 ip-10-0-129-205 kubenswrapper[2573]: I0909 20:59:50.123747    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9" podStartSLOduration=2.58092112 podStartE2EDuration="5.12373188s" totalImagesPullingTime="2.54281076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:50.123505405 +0000 UTC m=+462.334049189" watchObservedRunningTime="2026-09-09 20:59:50.12373188 +0000 UTC m=+462.334275664"
Sep 09 21:00:21.111916 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:00:21.111884    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-8tfg9"
Sep 09 21:01:04.720604 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.720511    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-2vwfl"]
Sep 09 21:01:04.722597 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.722581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:04.724846 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.724829    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 21:01:04.724969 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.724950    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4kpjr\""
Sep 09 21:01:04.738287 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.738263    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2vwfl"]
Sep 09 21:01:04.817718 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.817678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdb300b-1585-4468-912b-57d63e47cdef-tls-certs\") pod \"model-serving-api-7b954ffc95-2vwfl\" (UID: \"7bdb300b-1585-4468-912b-57d63e47cdef\") " pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:04.817718 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.817722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7pql\" (UniqueName: \"kubernetes.io/projected/7bdb300b-1585-4468-912b-57d63e47cdef-kube-api-access-t7pql\") pod \"model-serving-api-7b954ffc95-2vwfl\" (UID: \"7bdb300b-1585-4468-912b-57d63e47cdef\") " pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:04.919084 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.919044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdb300b-1585-4468-912b-57d63e47cdef-tls-certs\") pod \"model-serving-api-7b954ffc95-2vwfl\" (UID: \"7bdb300b-1585-4468-912b-57d63e47cdef\") " pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:04.919254 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.919096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7pql\" (UniqueName: \"kubernetes.io/projected/7bdb300b-1585-4468-912b-57d63e47cdef-kube-api-access-t7pql\") pod \"model-serving-api-7b954ffc95-2vwfl\" (UID: \"7bdb300b-1585-4468-912b-57d63e47cdef\") " pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:04.919254 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:01:04.919198    2573 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 21:01:04.919324 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:01:04.919278    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bdb300b-1585-4468-912b-57d63e47cdef-tls-certs podName:7bdb300b-1585-4468-912b-57d63e47cdef nodeName:}" failed. No retries permitted until 2026-09-09 21:01:05.419258229 +0000 UTC m=+537.629801993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7bdb300b-1585-4468-912b-57d63e47cdef-tls-certs") pod "model-serving-api-7b954ffc95-2vwfl" (UID: "7bdb300b-1585-4468-912b-57d63e47cdef") : secret "model-serving-api-tls" not found
Sep 09 21:01:04.928176 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:04.928143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7pql\" (UniqueName: \"kubernetes.io/projected/7bdb300b-1585-4468-912b-57d63e47cdef-kube-api-access-t7pql\") pod \"model-serving-api-7b954ffc95-2vwfl\" (UID: \"7bdb300b-1585-4468-912b-57d63e47cdef\") " pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:05.422876 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:05.422837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdb300b-1585-4468-912b-57d63e47cdef-tls-certs\") pod \"model-serving-api-7b954ffc95-2vwfl\" (UID: \"7bdb300b-1585-4468-912b-57d63e47cdef\") " pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:05.425225 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:05.425199    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdb300b-1585-4468-912b-57d63e47cdef-tls-certs\") pod \"model-serving-api-7b954ffc95-2vwfl\" (UID: \"7bdb300b-1585-4468-912b-57d63e47cdef\") " pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:05.632081 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:05.632026    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:05.902676 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:05.902646    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2vwfl"]
Sep 09 21:01:05.905427 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:01:05.905401    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdb300b_1585_4468_912b_57d63e47cdef.slice/crio-8a223b35950e7e1326b7a12254c8bdf4e032fcf5ef2f09a69bfbf93a3733ec84 WatchSource:0}: Error finding container 8a223b35950e7e1326b7a12254c8bdf4e032fcf5ef2f09a69bfbf93a3733ec84: Status 404 returned error can't find the container with id 8a223b35950e7e1326b7a12254c8bdf4e032fcf5ef2f09a69bfbf93a3733ec84
Sep 09 21:01:06.386190 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:06.386155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2vwfl" event={"ID":"7bdb300b-1585-4468-912b-57d63e47cdef","Type":"ContainerStarted","Data":"8a223b35950e7e1326b7a12254c8bdf4e032fcf5ef2f09a69bfbf93a3733ec84"}
Sep 09 21:01:08.396250 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:08.396204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2vwfl" event={"ID":"7bdb300b-1585-4468-912b-57d63e47cdef","Type":"ContainerStarted","Data":"a79d4ec821278cafb5294dc440be16ed90f5335a713bfaf6eb3176fa5a5ca1cf"}
Sep 09 21:01:08.396687 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:08.396363    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:01:08.424884 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:08.424829    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-2vwfl" podStartSLOduration=2.936961357 podStartE2EDuration="4.424813417s" totalImagesPullingTime="1.48785206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:08.423518245 +0000 UTC m=+540.634062022" watchObservedRunningTime="2026-09-09 21:01:08.424813417 +0000 UTC m=+540.635357201"
Sep 09 21:01:19.404416 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:01:19.404378    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-2vwfl"
Sep 09 21:02:08.245716 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:08.245686    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:02:08.246635 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:08.246614    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:02:52.708408 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.708375    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"]
Sep 09 21:02:52.711594 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.711576    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.713995 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.713973    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 09 21:02:52.713995 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.713988    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:02:52.714165 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.713979    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 21:02:52.714165 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.713980    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:02:52.714352 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.714314    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:02:52.714491 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.714458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:02:52.714533 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.714516    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:02:52.720253 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.720235    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"]
Sep 09 21:02:52.817022 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.816976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr54m\" (UniqueName: \"kubernetes.io/projected/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-api-access-hr54m\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.817022 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.817024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.817249 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.817052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.817249 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.817121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7962ae4-d359-456e-a60c-4ff005b2f768-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.817249 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.817181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918063 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918305 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918083    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr54m\" (UniqueName: \"kubernetes.io/projected/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-api-access-hr54m\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918305 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918113    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918305 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:02:52.918228    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 09 21:02:52.918305 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918305 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:02:52.918302    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls podName:e7962ae4-d359-456e-a60c-4ff005b2f768 nodeName:}" failed. No retries permitted until 2026-09-09 21:02:53.418282207 +0000 UTC m=+645.628825983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls") pod "isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" (UID: "e7962ae4-d359-456e-a60c-4ff005b2f768") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 09 21:02:52.918664 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7962ae4-d359-456e-a60c-4ff005b2f768-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918741 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918720    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7962ae4-d359-456e-a60c-4ff005b2f768-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918797 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.918851 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.918833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:52.926446 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:52.926422    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr54m\" (UniqueName: \"kubernetes.io/projected/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-api-access-hr54m\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:53.420817 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:53.420773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:53.423443 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:53.423420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:53.622459 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:53.622416    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:02:53.895124 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:53.895089    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"]
Sep 09 21:02:53.899278 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:02:53.899244    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7962ae4_d359_456e_a60c_4ff005b2f768.slice/crio-37656941bec65121c74d825b8dd123317c78bac24bcce8132c904ce3d76487c4 WatchSource:0}: Error finding container 37656941bec65121c74d825b8dd123317c78bac24bcce8132c904ce3d76487c4: Status 404 returned error can't find the container with id 37656941bec65121c74d825b8dd123317c78bac24bcce8132c904ce3d76487c4
Sep 09 21:02:53.901328 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:53.901310    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:02:54.793352 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:54.793300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerStarted","Data":"37656941bec65121c74d825b8dd123317c78bac24bcce8132c904ce3d76487c4"}
Sep 09 21:02:57.808656 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:02:57.808614    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerStarted","Data":"9a32e2fa1af6db45ef526778b32187461016a267ac05f96abe7fe6f483230587"}
Sep 09 21:03:01.824891 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:01.824860    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerID="9a32e2fa1af6db45ef526778b32187461016a267ac05f96abe7fe6f483230587" exitCode=0
Sep 09 21:03:01.825258 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:01.824915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerDied","Data":"9a32e2fa1af6db45ef526778b32187461016a267ac05f96abe7fe6f483230587"}
Sep 09 21:03:15.893918 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:15.893880    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerStarted","Data":"430183a1c248468b29f59b1d74101ae9fdc1fc94c6a69d1d8c1e3d5c0ab6fd03"}
Sep 09 21:03:18.910811 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:18.910778    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerStarted","Data":"caa06c007ae83d3b61391780c025091cbf74604fc5a866138bd85d8e53fec32f"}
Sep 09 21:03:18.911329 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:18.911002    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:03:18.911329 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:18.911156    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:03:18.912385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:18.912362    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:03:18.944258 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:18.944205    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podStartSLOduration=3.666560417 podStartE2EDuration="26.944189679s" totalImagesPullingTime="19.604154004s" totalInitContainerRuntime="3.673475258s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:18.942713277 +0000 UTC m=+671.153257071" watchObservedRunningTime="2026-09-09 21:03:18.944189679 +0000 UTC m=+671.154733462"
Sep 09 21:03:19.914730 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:19.914691    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:03:24.919344 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:24.919312    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:03:24.919862 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:24.919835    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:03:34.920673 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:34.920632    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:03:44.920627 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:44.920591    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:03:54.923828 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:03:54.923782    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:04:04.920341 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:04.920259    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:04:14.919803 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:14.919762    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:04:24.920133 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:24.920090    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:04:34.920725 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:34.920692    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:04:42.580109 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:42.580065    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"]
Sep 09 21:04:42.580579 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:42.580375    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" containerID="cri-o://430183a1c248468b29f59b1d74101ae9fdc1fc94c6a69d1d8c1e3d5c0ab6fd03" gracePeriod=30
Sep 09 21:04:42.580579 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:42.580414    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kube-rbac-proxy" containerID="cri-o://caa06c007ae83d3b61391780c025091cbf74604fc5a866138bd85d8e53fec32f" gracePeriod=30
Sep 09 21:04:43.219454 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:43.219412    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerID="caa06c007ae83d3b61391780c025091cbf74604fc5a866138bd85d8e53fec32f" exitCode=2
Sep 09 21:04:43.219658 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:43.219481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerDied","Data":"caa06c007ae83d3b61391780c025091cbf74604fc5a866138bd85d8e53fec32f"}
Sep 09 21:04:44.915962 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:44.915923    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 09 21:04:44.920273 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:44.920247    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 21:04:47.238180 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.238143    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerID="430183a1c248468b29f59b1d74101ae9fdc1fc94c6a69d1d8c1e3d5c0ab6fd03" exitCode=0
Sep 09 21:04:47.238587 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.238218    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerDied","Data":"430183a1c248468b29f59b1d74101ae9fdc1fc94c6a69d1d8c1e3d5c0ab6fd03"}
Sep 09 21:04:47.321830 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.321807    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:04:47.332866 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.332843    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr54m\" (UniqueName: \"kubernetes.io/projected/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-api-access-hr54m\") pod \"e7962ae4-d359-456e-a60c-4ff005b2f768\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") "
Sep 09 21:04:47.332986 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.332875    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls\") pod \"e7962ae4-d359-456e-a60c-4ff005b2f768\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") "
Sep 09 21:04:47.332986 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.332930    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-cabundle-cert\") pod \"e7962ae4-d359-456e-a60c-4ff005b2f768\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") "
Sep 09 21:04:47.332986 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.332959    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7962ae4-d359-456e-a60c-4ff005b2f768-kserve-provision-location\") pod \"e7962ae4-d359-456e-a60c-4ff005b2f768\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") "
Sep 09 21:04:47.333092 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.332994    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-rbac-proxy-sar-config\") pod \"e7962ae4-d359-456e-a60c-4ff005b2f768\" (UID: \"e7962ae4-d359-456e-a60c-4ff005b2f768\") "
Sep 09 21:04:47.333382 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.333349    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7962ae4-d359-456e-a60c-4ff005b2f768-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7962ae4-d359-456e-a60c-4ff005b2f768" (UID: "e7962ae4-d359-456e-a60c-4ff005b2f768"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:47.333593 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.333356    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7962ae4-d359-456e-a60c-4ff005b2f768" (UID: "e7962ae4-d359-456e-a60c-4ff005b2f768"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:47.333593 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.333419    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7962ae4-d359-456e-a60c-4ff005b2f768" (UID: "e7962ae4-d359-456e-a60c-4ff005b2f768"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:47.335230 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.335211    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-api-access-hr54m" (OuterVolumeSpecName: "kube-api-access-hr54m") pod "e7962ae4-d359-456e-a60c-4ff005b2f768" (UID: "e7962ae4-d359-456e-a60c-4ff005b2f768"). InnerVolumeSpecName "kube-api-access-hr54m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:47.335406 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.335390    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7962ae4-d359-456e-a60c-4ff005b2f768" (UID: "e7962ae4-d359-456e-a60c-4ff005b2f768"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:47.433807 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.433718    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:04:47.433807 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.433749    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr54m\" (UniqueName: \"kubernetes.io/projected/e7962ae4-d359-456e-a60c-4ff005b2f768-kube-api-access-hr54m\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:04:47.433807 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.433759    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7962ae4-d359-456e-a60c-4ff005b2f768-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:04:47.433807 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.433768    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7962ae4-d359-456e-a60c-4ff005b2f768-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:04:47.433807 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.433775    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7962ae4-d359-456e-a60c-4ff005b2f768-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:04:47.976724 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.976689    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"]
Sep 09 21:04:47.976974 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.976962    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kube-rbac-proxy"
Sep 09 21:04:47.977021 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.976975    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kube-rbac-proxy"
Sep 09 21:04:47.977021 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.976994    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="storage-initializer"
Sep 09 21:04:47.977021 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.976999    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="storage-initializer"
Sep 09 21:04:47.977021 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.977008    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container"
Sep 09 21:04:47.977021 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.977014    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container"
Sep 09 21:04:47.977166 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.977060    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kserve-container"
Sep 09 21:04:47.977166 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.977067    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" containerName="kube-rbac-proxy"
Sep 09 21:04:47.979361 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.979345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:47.982661 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.982613    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c959c-predictor-serving-cert\""
Sep 09 21:04:47.982882 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:47.982867    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c959c-kube-rbac-proxy-sar-config\""
Sep 09 21:04:48.001085 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.001061    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"]
Sep 09 21:04:48.038272 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.038246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.038399 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.038284    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6411eef3-32eb-4fdb-9e34-026aa507496a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.038399 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.038304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjpzk\" (UniqueName: \"kubernetes.io/projected/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-api-access-sjpzk\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.038399 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.038324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.038533 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.038404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6411eef3-32eb-4fdb-9e34-026aa507496a-proxy-tls\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.139363 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.139319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.139619 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.139385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6411eef3-32eb-4fdb-9e34-026aa507496a-proxy-tls\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.139619 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.139419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.139619 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.139453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6411eef3-32eb-4fdb-9e34-026aa507496a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.139619 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.139486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjpzk\" (UniqueName: \"kubernetes.io/projected/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-api-access-sjpzk\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.139899 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.139874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6411eef3-32eb-4fdb-9e34-026aa507496a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.140060 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.140035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.140129 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.140108    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.142003 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.141987    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6411eef3-32eb-4fdb-9e34-026aa507496a-proxy-tls\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.167494 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.167460    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjpzk\" (UniqueName: \"kubernetes.io/projected/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-api-access-sjpzk\") pod \"isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.244134 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.244060    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"
Sep 09 21:04:48.244134 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.244078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr" event={"ID":"e7962ae4-d359-456e-a60c-4ff005b2f768","Type":"ContainerDied","Data":"37656941bec65121c74d825b8dd123317c78bac24bcce8132c904ce3d76487c4"}
Sep 09 21:04:48.244609 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.244139    2573 scope.go:117] "RemoveContainer" containerID="caa06c007ae83d3b61391780c025091cbf74604fc5a866138bd85d8e53fec32f"
Sep 09 21:04:48.253084 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.253048    2573 scope.go:117] "RemoveContainer" containerID="430183a1c248468b29f59b1d74101ae9fdc1fc94c6a69d1d8c1e3d5c0ab6fd03"
Sep 09 21:04:48.254272 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.254254    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"]
Sep 09 21:04:48.257819 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.257801    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.260891 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.260871    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c959c-kube-rbac-proxy-sar-config\""
Sep 09 21:04:48.261143 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.261127    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c959c-predictor-serving-cert\""
Sep 09 21:04:48.261203 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.261153    2573 scope.go:117] "RemoveContainer" containerID="9a32e2fa1af6db45ef526778b32187461016a267ac05f96abe7fe6f483230587"
Sep 09 21:04:48.275667 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.275644    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"]
Sep 09 21:04:48.287439 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.287418    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"]
Sep 09 21:04:48.287570 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.287478    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6df64bc5fd-7l4pr"]
Sep 09 21:04:48.288576 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.288534    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:48.340742 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.340571    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e302a478-cb81-448e-a826-a787c5bbb351-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.340742 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.340650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e302a478-cb81-448e-a826-a787c5bbb351-proxy-tls\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.340742 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.340680    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.340742 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.340707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.340986 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.340800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdcg\" (UniqueName: \"kubernetes.io/projected/e302a478-cb81-448e-a826-a787c5bbb351-kube-api-access-qsdcg\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.441265 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.441219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e302a478-cb81-448e-a826-a787c5bbb351-proxy-tls\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.441437 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.441276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.441437 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.441321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.441437 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.441411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdcg\" (UniqueName: \"kubernetes.io/projected/e302a478-cb81-448e-a826-a787c5bbb351-kube-api-access-qsdcg\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.441658 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.441451    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e302a478-cb81-448e-a826-a787c5bbb351-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.441894 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.441865    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e302a478-cb81-448e-a826-a787c5bbb351-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.442150 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.442097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.442243 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.442187    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.444018 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.443996    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e302a478-cb81-448e-a826-a787c5bbb351-proxy-tls\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.452786 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.452762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdcg\" (UniqueName: \"kubernetes.io/projected/e302a478-cb81-448e-a826-a787c5bbb351-kube-api-access-qsdcg\") pod \"isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.567530 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.567502    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:04:48.568908 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.568876    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"]
Sep 09 21:04:48.572648 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:04:48.572612    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6411eef3_32eb_4fdb_9e34_026aa507496a.slice/crio-36fc71ce163d95f4266f280461fbca6bd310f6a269d300a031f07d66fd085894 WatchSource:0}: Error finding container 36fc71ce163d95f4266f280461fbca6bd310f6a269d300a031f07d66fd085894: Status 404 returned error can't find the container with id 36fc71ce163d95f4266f280461fbca6bd310f6a269d300a031f07d66fd085894
Sep 09 21:04:48.876247 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:48.876171    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"]
Sep 09 21:04:48.879700 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:04:48.879673    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode302a478_cb81_448e_a826_a787c5bbb351.slice/crio-ed842fcd2a95d2294c3c5dd0cfaab0a7f39d9d45a0ff49edad217d8b5ecde71f WatchSource:0}: Error finding container ed842fcd2a95d2294c3c5dd0cfaab0a7f39d9d45a0ff49edad217d8b5ecde71f: Status 404 returned error can't find the container with id ed842fcd2a95d2294c3c5dd0cfaab0a7f39d9d45a0ff49edad217d8b5ecde71f
Sep 09 21:04:49.250607 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:49.250505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerStarted","Data":"5cb5da01dd22240e6657cec0543ffb419eeddb5c81e2bf4b6964b69ba45b7ea6"}
Sep 09 21:04:49.250607 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:49.250598    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerStarted","Data":"36fc71ce163d95f4266f280461fbca6bd310f6a269d300a031f07d66fd085894"}
Sep 09 21:04:49.253146 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:49.253119    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerStarted","Data":"8f0549f236465114382b83dadb085d3202672105f5ec2aeefd2f6308dccd7e00"}
Sep 09 21:04:49.253268 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:49.253171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerStarted","Data":"ed842fcd2a95d2294c3c5dd0cfaab0a7f39d9d45a0ff49edad217d8b5ecde71f"}
Sep 09 21:04:50.287367 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:50.287332    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7962ae4-d359-456e-a60c-4ff005b2f768" path="/var/lib/kubelet/pods/e7962ae4-d359-456e-a60c-4ff005b2f768/volumes"
Sep 09 21:04:52.267905 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:52.267878    2573 generic.go:356] "Generic (PLEG): container finished" podID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerID="5cb5da01dd22240e6657cec0543ffb419eeddb5c81e2bf4b6964b69ba45b7ea6" exitCode=0
Sep 09 21:04:52.268190 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:52.267946    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerDied","Data":"5cb5da01dd22240e6657cec0543ffb419eeddb5c81e2bf4b6964b69ba45b7ea6"}
Sep 09 21:04:53.273974 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:53.273892    2573 generic.go:356] "Generic (PLEG): container finished" podID="e302a478-cb81-448e-a826-a787c5bbb351" containerID="8f0549f236465114382b83dadb085d3202672105f5ec2aeefd2f6308dccd7e00" exitCode=0
Sep 09 21:04:53.273974 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:53.273964    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerDied","Data":"8f0549f236465114382b83dadb085d3202672105f5ec2aeefd2f6308dccd7e00"}
Sep 09 21:04:53.276239 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:53.276216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerStarted","Data":"13a94158086263af77f46a0dacfeb8d578711283087f43f90bb7179251d4536f"}
Sep 09 21:04:53.276331 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:53.276272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerStarted","Data":"639a985ee6f05843ab43a276ce5934fd14695fba11778307a404be5b0c479ab5"}
Sep 09 21:04:53.276474 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:53.276459    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:53.314220 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:53.314163    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podStartSLOduration=2.689121547 podStartE2EDuration="6.314142095s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.625020548s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:53.312192463 +0000 UTC m=+765.522736273" watchObservedRunningTime="2026-09-09 21:04:53.314142095 +0000 UTC m=+765.524685880"
Sep 09 21:04:54.283889 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:54.283853    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:04:54.289071 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:54.289033    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:04:55.288844 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:04:55.288799    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:05:00.298777 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:00.298748    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:05:00.299380 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:00.299356    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:05:10.300331 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:10.300279    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:05:14.381438 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:14.381397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerStarted","Data":"424e9f015499059bf145f557e813dd38d30f62e9bc6725c24e15ffea1e049158"}
Sep 09 21:05:14.381841 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:14.381506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerStarted","Data":"36e4b1167c88d7e88ac951eec52dfdd9f01c49f574324f95d65e2914473e7ce3"}
Sep 09 21:05:14.381841 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:14.381820    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:05:14.406409 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:14.406361    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podStartSLOduration=2.164318662 podStartE2EDuration="26.406345738s" totalImagesPullingTime="20.296853939s" totalInitContainerRuntime="3.945173137s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:14.406070673 +0000 UTC m=+786.616614458" watchObservedRunningTime="2026-09-09 21:05:14.406345738 +0000 UTC m=+786.616889523"
Sep 09 21:05:15.385503 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:15.385470    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:05:15.386870 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:15.386840    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:05:16.389512 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:16.389470    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:05:20.299930 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:20.299886    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:05:21.393708 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:21.393680    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:05:21.394259 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:21.394229    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:05:30.300149 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:30.300069    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:05:31.394819 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:31.394773    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:05:40.299732 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:40.299693    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:05:41.394706 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:41.394662    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:05:50.299546 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:50.299507    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:05:51.394207 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:05:51.394165    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:06:00.299908 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:00.299869    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:06:01.394771 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:01.394733    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:06:10.300723 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:10.300695    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:06:11.394182 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:11.394141    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:06:21.395739 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:21.395702    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:06:38.055102 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.055070    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"]
Sep 09 21:06:38.055547 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.055310    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" containerID="cri-o://639a985ee6f05843ab43a276ce5934fd14695fba11778307a404be5b0c479ab5" gracePeriod=30
Sep 09 21:06:38.055547 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.055377    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kube-rbac-proxy" containerID="cri-o://13a94158086263af77f46a0dacfeb8d578711283087f43f90bb7179251d4536f" gracePeriod=30
Sep 09 21:06:38.090938 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.090908    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"]
Sep 09 21:06:38.091195 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.091153    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" containerID="cri-o://36e4b1167c88d7e88ac951eec52dfdd9f01c49f574324f95d65e2914473e7ce3" gracePeriod=30
Sep 09 21:06:38.091318 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.091219    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kube-rbac-proxy" containerID="cri-o://424e9f015499059bf145f557e813dd38d30f62e9bc6725c24e15ffea1e049158" gracePeriod=30
Sep 09 21:06:38.725224 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.725189    2573 generic.go:356] "Generic (PLEG): container finished" podID="e302a478-cb81-448e-a826-a787c5bbb351" containerID="424e9f015499059bf145f557e813dd38d30f62e9bc6725c24e15ffea1e049158" exitCode=2
Sep 09 21:06:38.725390 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.725264    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerDied","Data":"424e9f015499059bf145f557e813dd38d30f62e9bc6725c24e15ffea1e049158"}
Sep 09 21:06:38.727208 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.727185    2573 generic.go:356] "Generic (PLEG): container finished" podID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerID="13a94158086263af77f46a0dacfeb8d578711283087f43f90bb7179251d4536f" exitCode=2
Sep 09 21:06:38.727334 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:38.727225    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerDied","Data":"13a94158086263af77f46a0dacfeb8d578711283087f43f90bb7179251d4536f"}
Sep 09 21:06:40.289734 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:40.289694    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 09 21:06:40.300168 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:40.300132    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 21:06:41.389721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.389684    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 21:06:41.395069 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.395034    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 21:06:41.741895 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.741864    2573 generic.go:356] "Generic (PLEG): container finished" podID="e302a478-cb81-448e-a826-a787c5bbb351" containerID="36e4b1167c88d7e88ac951eec52dfdd9f01c49f574324f95d65e2914473e7ce3" exitCode=0
Sep 09 21:06:41.742064 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.741936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerDied","Data":"36e4b1167c88d7e88ac951eec52dfdd9f01c49f574324f95d65e2914473e7ce3"}
Sep 09 21:06:41.832419 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.832394    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:06:41.927155 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927069    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-kube-rbac-proxy-sar-config\") pod \"e302a478-cb81-448e-a826-a787c5bbb351\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") "
Sep 09 21:06:41.927155 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927127    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e302a478-cb81-448e-a826-a787c5bbb351-proxy-tls\") pod \"e302a478-cb81-448e-a826-a787c5bbb351\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") "
Sep 09 21:06:41.927354 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927168    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e302a478-cb81-448e-a826-a787c5bbb351-kserve-provision-location\") pod \"e302a478-cb81-448e-a826-a787c5bbb351\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") "
Sep 09 21:06:41.927354 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927203    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsdcg\" (UniqueName: \"kubernetes.io/projected/e302a478-cb81-448e-a826-a787c5bbb351-kube-api-access-qsdcg\") pod \"e302a478-cb81-448e-a826-a787c5bbb351\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") "
Sep 09 21:06:41.927354 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927230    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-cabundle-cert\") pod \"e302a478-cb81-448e-a826-a787c5bbb351\" (UID: \"e302a478-cb81-448e-a826-a787c5bbb351\") "
Sep 09 21:06:41.927481 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927449    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e302a478-cb81-448e-a826-a787c5bbb351" (UID: "e302a478-cb81-448e-a826-a787c5bbb351"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:41.927607 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927574    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e302a478-cb81-448e-a826-a787c5bbb351-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e302a478-cb81-448e-a826-a787c5bbb351" (UID: "e302a478-cb81-448e-a826-a787c5bbb351"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:41.927740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.927652    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e302a478-cb81-448e-a826-a787c5bbb351" (UID: "e302a478-cb81-448e-a826-a787c5bbb351"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:41.929393 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.929371    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e302a478-cb81-448e-a826-a787c5bbb351-kube-api-access-qsdcg" (OuterVolumeSpecName: "kube-api-access-qsdcg") pod "e302a478-cb81-448e-a826-a787c5bbb351" (UID: "e302a478-cb81-448e-a826-a787c5bbb351"). InnerVolumeSpecName "kube-api-access-qsdcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:41.929492 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:41.929397    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e302a478-cb81-448e-a826-a787c5bbb351-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e302a478-cb81-448e-a826-a787c5bbb351" (UID: "e302a478-cb81-448e-a826-a787c5bbb351"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:42.027933 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.027895    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:42.027933 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.027926    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e302a478-cb81-448e-a826-a787c5bbb351-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:42.027933 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.027936    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e302a478-cb81-448e-a826-a787c5bbb351-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:42.028165 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.027946    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsdcg\" (UniqueName: \"kubernetes.io/projected/e302a478-cb81-448e-a826-a787c5bbb351-kube-api-access-qsdcg\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:42.028165 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.027954    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e302a478-cb81-448e-a826-a787c5bbb351-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:42.749689 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.749650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz" event={"ID":"e302a478-cb81-448e-a826-a787c5bbb351","Type":"ContainerDied","Data":"ed842fcd2a95d2294c3c5dd0cfaab0a7f39d9d45a0ff49edad217d8b5ecde71f"}
Sep 09 21:06:42.749689 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.749678    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"
Sep 09 21:06:42.750237 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.749750    2573 scope.go:117] "RemoveContainer" containerID="424e9f015499059bf145f557e813dd38d30f62e9bc6725c24e15ffea1e049158"
Sep 09 21:06:42.752918 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.752889    2573 generic.go:356] "Generic (PLEG): container finished" podID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerID="639a985ee6f05843ab43a276ce5934fd14695fba11778307a404be5b0c479ab5" exitCode=0
Sep 09 21:06:42.753046 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.752956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerDied","Data":"639a985ee6f05843ab43a276ce5934fd14695fba11778307a404be5b0c479ab5"}
Sep 09 21:06:42.758698 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.758681    2573 scope.go:117] "RemoveContainer" containerID="36e4b1167c88d7e88ac951eec52dfdd9f01c49f574324f95d65e2914473e7ce3"
Sep 09 21:06:42.766265 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.766236    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"]
Sep 09 21:06:42.767084 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.767066    2573 scope.go:117] "RemoveContainer" containerID="8f0549f236465114382b83dadb085d3202672105f5ec2aeefd2f6308dccd7e00"
Sep 09 21:06:42.771043 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.771018    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c959c-predictor-74899bfbdf-w4tvz"]
Sep 09 21:06:42.801205 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.801185    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:06:42.936421 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936333    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6411eef3-32eb-4fdb-9e34-026aa507496a-proxy-tls\") pod \"6411eef3-32eb-4fdb-9e34-026aa507496a\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") "
Sep 09 21:06:42.936421 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6411eef3-32eb-4fdb-9e34-026aa507496a-kserve-provision-location\") pod \"6411eef3-32eb-4fdb-9e34-026aa507496a\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") "
Sep 09 21:06:42.936421 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936411    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-cabundle-cert\") pod \"6411eef3-32eb-4fdb-9e34-026aa507496a\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") "
Sep 09 21:06:42.936717 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936432    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-rbac-proxy-sar-config\") pod \"6411eef3-32eb-4fdb-9e34-026aa507496a\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") "
Sep 09 21:06:42.936717 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936453    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjpzk\" (UniqueName: \"kubernetes.io/projected/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-api-access-sjpzk\") pod \"6411eef3-32eb-4fdb-9e34-026aa507496a\" (UID: \"6411eef3-32eb-4fdb-9e34-026aa507496a\") "
Sep 09 21:06:42.936820 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936788    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6411eef3-32eb-4fdb-9e34-026aa507496a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6411eef3-32eb-4fdb-9e34-026aa507496a" (UID: "6411eef3-32eb-4fdb-9e34-026aa507496a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:42.936820 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936796    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6411eef3-32eb-4fdb-9e34-026aa507496a" (UID: "6411eef3-32eb-4fdb-9e34-026aa507496a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:42.936928 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.936819    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6411eef3-32eb-4fdb-9e34-026aa507496a" (UID: "6411eef3-32eb-4fdb-9e34-026aa507496a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:42.938761 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.938741    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-api-access-sjpzk" (OuterVolumeSpecName: "kube-api-access-sjpzk") pod "6411eef3-32eb-4fdb-9e34-026aa507496a" (UID: "6411eef3-32eb-4fdb-9e34-026aa507496a"). InnerVolumeSpecName "kube-api-access-sjpzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:42.938761 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:42.938753    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6411eef3-32eb-4fdb-9e34-026aa507496a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6411eef3-32eb-4fdb-9e34-026aa507496a" (UID: "6411eef3-32eb-4fdb-9e34-026aa507496a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:43.037740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.037702    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:43.037740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.037734    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:43.037740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.037744    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjpzk\" (UniqueName: \"kubernetes.io/projected/6411eef3-32eb-4fdb-9e34-026aa507496a-kube-api-access-sjpzk\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:43.037959 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.037753    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6411eef3-32eb-4fdb-9e34-026aa507496a-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:43.037959 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.037762    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6411eef3-32eb-4fdb-9e34-026aa507496a-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:06:43.758805 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.758778    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"
Sep 09 21:06:43.759269 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.758814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb" event={"ID":"6411eef3-32eb-4fdb-9e34-026aa507496a","Type":"ContainerDied","Data":"36fc71ce163d95f4266f280461fbca6bd310f6a269d300a031f07d66fd085894"}
Sep 09 21:06:43.759269 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.758883    2573 scope.go:117] "RemoveContainer" containerID="13a94158086263af77f46a0dacfeb8d578711283087f43f90bb7179251d4536f"
Sep 09 21:06:43.767824 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.767795    2573 scope.go:117] "RemoveContainer" containerID="639a985ee6f05843ab43a276ce5934fd14695fba11778307a404be5b0c479ab5"
Sep 09 21:06:43.775577 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.775535    2573 scope.go:117] "RemoveContainer" containerID="5cb5da01dd22240e6657cec0543ffb419eeddb5c81e2bf4b6964b69ba45b7ea6"
Sep 09 21:06:43.780879 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.780858    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"]
Sep 09 21:06:43.785214 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:43.785191    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c959c-predictor-bd4d96ffd-b8xhb"]
Sep 09 21:06:44.292190 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:44.292150    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" path="/var/lib/kubelet/pods/6411eef3-32eb-4fdb-9e34-026aa507496a/volumes"
Sep 09 21:06:44.292924 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:06:44.292907    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e302a478-cb81-448e-a826-a787c5bbb351" path="/var/lib/kubelet/pods/e302a478-cb81-448e-a826-a787c5bbb351/volumes"
Sep 09 21:07:08.268304 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:08.268224    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:07:08.270697 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:08.270674    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:07:58.238060 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238025    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"]
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238290    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kube-rbac-proxy"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238300    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kube-rbac-proxy"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238311    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="storage-initializer"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238316    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="storage-initializer"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238326    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238331    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238339    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238343    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238352    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="storage-initializer"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238357    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="storage-initializer"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238362    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kube-rbac-proxy"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238367    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kube-rbac-proxy"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238410    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kserve-container"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238421    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6411eef3-32eb-4fdb-9e34-026aa507496a" containerName="kube-rbac-proxy"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238427    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kserve-container"
Sep 09 21:07:58.240531 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.238433    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e302a478-cb81-448e-a826-a787c5bbb351" containerName="kube-rbac-proxy"
Sep 09 21:07:58.241334 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.241320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.243635 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.243607    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:07:58.243752 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.243672    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ea23b7-kube-rbac-proxy-sar-config\""
Sep 09 21:07:58.244336 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.244316    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ea23b7-predictor-serving-cert\""
Sep 09 21:07:58.244422 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.244357    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:07:58.244422 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.244399    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:07:58.244422 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.244408    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:07:58.244422 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.244358    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:07:58.252190 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.252168    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"]
Sep 09 21:07:58.309808 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.309775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.309971 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.309813    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22d528ab-0637-42cf-b7a9-61e65cf99a15-kserve-provision-location\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.309971 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.309834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97f7r\" (UniqueName: \"kubernetes.io/projected/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-api-access-97f7r\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.309971 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.309944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.310075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.309982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-cabundle-cert\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.410667 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.410637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.410667 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.410669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-cabundle-cert\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.410955 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.410723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.410955 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.410745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22d528ab-0637-42cf-b7a9-61e65cf99a15-kserve-provision-location\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.410955 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.410770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97f7r\" (UniqueName: \"kubernetes.io/projected/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-api-access-97f7r\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.410955 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:07:58.410783    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-serving-cert: secret "isvc-primary-ea23b7-predictor-serving-cert" not found
Sep 09 21:07:58.410955 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:07:58.410872    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls podName:22d528ab-0637-42cf-b7a9-61e65cf99a15 nodeName:}" failed. No retries permitted until 2026-09-09 21:07:58.910851793 +0000 UTC m=+951.121395573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls") pod "isvc-primary-ea23b7-predictor-844b656767-t9mtg" (UID: "22d528ab-0637-42cf-b7a9-61e65cf99a15") : secret "isvc-primary-ea23b7-predictor-serving-cert" not found
Sep 09 21:07:58.411156 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.411140    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22d528ab-0637-42cf-b7a9-61e65cf99a15-kserve-provision-location\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.411440 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.411423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-cabundle-cert\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.411475 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.411423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.418889 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.418867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97f7r\" (UniqueName: \"kubernetes.io/projected/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-api-access-97f7r\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.914985 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.914945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:58.917473 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:58.917453    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls\") pod \"isvc-primary-ea23b7-predictor-844b656767-t9mtg\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") " pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:59.151290 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:59.151247    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:07:59.425136 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:07:59.425113    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"]
Sep 09 21:07:59.427806 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:07:59.427777    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d528ab_0637_42cf_b7a9_61e65cf99a15.slice/crio-885fb837b402c9c12b4278c4bdc939711e32dd0ef27d055b0005588cfe3ba86a WatchSource:0}: Error finding container 885fb837b402c9c12b4278c4bdc939711e32dd0ef27d055b0005588cfe3ba86a: Status 404 returned error can't find the container with id 885fb837b402c9c12b4278c4bdc939711e32dd0ef27d055b0005588cfe3ba86a
Sep 09 21:08:00.053465 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:00.053427    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerStarted","Data":"f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3"}
Sep 09 21:08:00.053678 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:00.053495    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerStarted","Data":"885fb837b402c9c12b4278c4bdc939711e32dd0ef27d055b0005588cfe3ba86a"}
Sep 09 21:08:04.070210 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:04.070173    2573 generic.go:356] "Generic (PLEG): container finished" podID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerID="f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3" exitCode=0
Sep 09 21:08:04.070584 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:04.070244    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerDied","Data":"f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3"}
Sep 09 21:08:05.075905 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:05.075872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerStarted","Data":"0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165"}
Sep 09 21:08:05.076352 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:05.075933    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerStarted","Data":"a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e"}
Sep 09 21:08:05.076352 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:05.076188    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:08:05.076352 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:05.076232    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:08:05.077672 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:05.077648    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:08:05.104454 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:05.104403    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podStartSLOduration=3.226631554 podStartE2EDuration="7.104385846s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877754292s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:05.103204082 +0000 UTC m=+957.313747880" watchObservedRunningTime="2026-09-09 21:08:05.104385846 +0000 UTC m=+957.314929678"
Sep 09 21:08:06.079483 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:06.079440    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:08:11.083874 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:11.083844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:08:11.084571 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:11.084514    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:08:21.084877 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:21.084829    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:08:31.085364 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:31.085285    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:08:41.085341 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:41.085303    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:08:51.084629 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:08:51.084583    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:09:01.084896 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:09:01.084857    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:09:11.085168 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:09:11.085127    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 21:09:21.085279 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:09:21.085250    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:10:01.478470 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:01.478389    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"]
Sep 09 21:10:01.479069 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:01.478645    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container" containerID="cri-o://a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e" gracePeriod=30
Sep 09 21:10:01.479069 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:01.478754    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kube-rbac-proxy" containerID="cri-o://0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165" gracePeriod=30
Sep 09 21:10:02.521010 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:02.520973    2573 generic.go:356] "Generic (PLEG): container finished" podID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerID="0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165" exitCode=2
Sep 09 21:10:02.521376 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:02.521048    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerDied","Data":"0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165"}
Sep 09 21:10:06.080657 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.080614    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 09 21:10:06.220582 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.220538    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:10:06.303068 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303039    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97f7r\" (UniqueName: \"kubernetes.io/projected/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-api-access-97f7r\") pod \"22d528ab-0637-42cf-b7a9-61e65cf99a15\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") "
Sep 09 21:10:06.303200 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303112    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-cabundle-cert\") pod \"22d528ab-0637-42cf-b7a9-61e65cf99a15\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") "
Sep 09 21:10:06.303200 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303142    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-rbac-proxy-sar-config\") pod \"22d528ab-0637-42cf-b7a9-61e65cf99a15\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") "
Sep 09 21:10:06.303200 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303162    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls\") pod \"22d528ab-0637-42cf-b7a9-61e65cf99a15\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") "
Sep 09 21:10:06.303200 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303181    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22d528ab-0637-42cf-b7a9-61e65cf99a15-kserve-provision-location\") pod \"22d528ab-0637-42cf-b7a9-61e65cf99a15\" (UID: \"22d528ab-0637-42cf-b7a9-61e65cf99a15\") "
Sep 09 21:10:06.303512 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303488    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22d528ab-0637-42cf-b7a9-61e65cf99a15" (UID: "22d528ab-0637-42cf-b7a9-61e65cf99a15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:06.303512 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303496    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22d528ab-0637-42cf-b7a9-61e65cf99a15" (UID: "22d528ab-0637-42cf-b7a9-61e65cf99a15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:06.303644 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.303526    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d528ab-0637-42cf-b7a9-61e65cf99a15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22d528ab-0637-42cf-b7a9-61e65cf99a15" (UID: "22d528ab-0637-42cf-b7a9-61e65cf99a15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:06.305314 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.305294    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22d528ab-0637-42cf-b7a9-61e65cf99a15" (UID: "22d528ab-0637-42cf-b7a9-61e65cf99a15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:06.305379 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.305338    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-api-access-97f7r" (OuterVolumeSpecName: "kube-api-access-97f7r") pod "22d528ab-0637-42cf-b7a9-61e65cf99a15" (UID: "22d528ab-0637-42cf-b7a9-61e65cf99a15"). InnerVolumeSpecName "kube-api-access-97f7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:06.404565 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.404523    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97f7r\" (UniqueName: \"kubernetes.io/projected/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-api-access-97f7r\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:06.404740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.404580    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:06.404740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.404590    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d528ab-0637-42cf-b7a9-61e65cf99a15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:06.404740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.404599    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d528ab-0637-42cf-b7a9-61e65cf99a15-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:06.404740 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.404607    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22d528ab-0637-42cf-b7a9-61e65cf99a15-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:06.536939 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.536895    2573 generic.go:356] "Generic (PLEG): container finished" podID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerID="a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e" exitCode=0
Sep 09 21:10:06.537123 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.536957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerDied","Data":"a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e"}
Sep 09 21:10:06.537123 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.537006    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg" event={"ID":"22d528ab-0637-42cf-b7a9-61e65cf99a15","Type":"ContainerDied","Data":"885fb837b402c9c12b4278c4bdc939711e32dd0ef27d055b0005588cfe3ba86a"}
Sep 09 21:10:06.537123 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.537031    2573 scope.go:117] "RemoveContainer" containerID="0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165"
Sep 09 21:10:06.537123 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.536959    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"
Sep 09 21:10:06.545572 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.545530    2573 scope.go:117] "RemoveContainer" containerID="a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e"
Sep 09 21:10:06.553017 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.552991    2573 scope.go:117] "RemoveContainer" containerID="f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3"
Sep 09 21:10:06.560298 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.560247    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"]
Sep 09 21:10:06.560940 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.560924    2573 scope.go:117] "RemoveContainer" containerID="0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165"
Sep 09 21:10:06.561220 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:10:06.561199    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165\": container with ID starting with 0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165 not found: ID does not exist" containerID="0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165"
Sep 09 21:10:06.561294 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.561230    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165"} err="failed to get container status \"0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165\": rpc error: code = NotFound desc = could not find container \"0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165\": container with ID starting with 0ba466d5ca42e3bd6dad428eb067892c6bf2d02a894aa0223625f04d690aa165 not found: ID does not exist"
Sep 09 21:10:06.561294 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.561270    2573 scope.go:117] "RemoveContainer" containerID="a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e"
Sep 09 21:10:06.561582 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:10:06.561546    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e\": container with ID starting with a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e not found: ID does not exist" containerID="a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e"
Sep 09 21:10:06.561655 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.561590    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e"} err="failed to get container status \"a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e\": rpc error: code = NotFound desc = could not find container \"a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e\": container with ID starting with a5da32ac654c25d76bb858216ce04411cc3ef43aab10d634a24d817bdfedd61e not found: ID does not exist"
Sep 09 21:10:06.561655 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.561618    2573 scope.go:117] "RemoveContainer" containerID="f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3"
Sep 09 21:10:06.561897 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:10:06.561878    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3\": container with ID starting with f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3 not found: ID does not exist" containerID="f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3"
Sep 09 21:10:06.561976 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.561904    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3"} err="failed to get container status \"f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3\": rpc error: code = NotFound desc = could not find container \"f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3\": container with ID starting with f053a414fd2ae8cd7eb208f7c298b39a94ff0f379cd089098052f121dcf8b4d3 not found: ID does not exist"
Sep 09 21:10:06.563280 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.563262    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ea23b7-predictor-844b656767-t9mtg"]
Sep 09 21:10:06.679155 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679120    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"]
Sep 09 21:10:06.679460 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679446    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="storage-initializer"
Sep 09 21:10:06.679502 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679466    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="storage-initializer"
Sep 09 21:10:06.679502 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679485    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kube-rbac-proxy"
Sep 09 21:10:06.679502 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679490    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kube-rbac-proxy"
Sep 09 21:10:06.679640 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679505    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container"
Sep 09 21:10:06.679640 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679511    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container"
Sep 09 21:10:06.679640 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679583    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kube-rbac-proxy"
Sep 09 21:10:06.679640 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.679594    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" containerName="kserve-container"
Sep 09 21:10:06.684191 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.684175    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.686448 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.686428    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:10:06.686546 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.686451    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2aef6d-predictor-serving-cert\""
Sep 09 21:10:06.686758 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.686744    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:10:06.686821 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.686755    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2aef6d-kube-rbac-proxy-sar-config\""
Sep 09 21:10:06.686876 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.686831    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:10:06.686913 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.686885    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-2aef6d-dockercfg-hcwjw\""
Sep 09 21:10:06.686956 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.686931    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-2aef6d\""
Sep 09 21:10:06.692079 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.692055    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"]
Sep 09 21:10:06.807383 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.807339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-cabundle-cert\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.807588 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.807391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.807588 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.807453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kserve-provision-location\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.807588 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.807478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bf8p\" (UniqueName: \"kubernetes.io/projected/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-api-access-4bf8p\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.807702 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.807580    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbf776a-2d18-4dd7-ae34-c442649f1fce-proxy-tls\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908012 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.907930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908012 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.907973    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kserve-provision-location\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908012 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.907997    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bf8p\" (UniqueName: \"kubernetes.io/projected/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-api-access-4bf8p\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908297 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.908049    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbf776a-2d18-4dd7-ae34-c442649f1fce-proxy-tls\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908297 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.908072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-cabundle-cert\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908455 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.908425    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kserve-provision-location\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908698 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.908679    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.908756 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.908688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-cabundle-cert\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.910629 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.910606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbf776a-2d18-4dd7-ae34-c442649f1fce-proxy-tls\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.917022 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.917001    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bf8p\" (UniqueName: \"kubernetes.io/projected/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-api-access-4bf8p\") pod \"isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") " pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:06.994731 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:06.994692    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:07.273655 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:07.273516    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"]
Sep 09 21:10:07.276039 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:10:07.276001    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbf776a_2d18_4dd7_ae34_c442649f1fce.slice/crio-2618ca6027fdfe34ee18582f4c40e877eeee8a0cedf54a93c38ef93957ae3b01 WatchSource:0}: Error finding container 2618ca6027fdfe34ee18582f4c40e877eeee8a0cedf54a93c38ef93957ae3b01: Status 404 returned error can't find the container with id 2618ca6027fdfe34ee18582f4c40e877eeee8a0cedf54a93c38ef93957ae3b01
Sep 09 21:10:07.542961 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:07.542928    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8" event={"ID":"1dbf776a-2d18-4dd7-ae34-c442649f1fce","Type":"ContainerStarted","Data":"f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470"}
Sep 09 21:10:07.543194 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:07.542988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8" event={"ID":"1dbf776a-2d18-4dd7-ae34-c442649f1fce","Type":"ContainerStarted","Data":"2618ca6027fdfe34ee18582f4c40e877eeee8a0cedf54a93c38ef93957ae3b01"}
Sep 09 21:10:08.287743 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:08.287707    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d528ab-0637-42cf-b7a9-61e65cf99a15" path="/var/lib/kubelet/pods/22d528ab-0637-42cf-b7a9-61e65cf99a15/volumes"
Sep 09 21:10:13.567849 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:13.567820    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8_1dbf776a-2d18-4dd7-ae34-c442649f1fce/storage-initializer/0.log"
Sep 09 21:10:13.568203 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:13.567857    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerID="f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470" exitCode=1
Sep 09 21:10:13.568203 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:13.567937    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8" event={"ID":"1dbf776a-2d18-4dd7-ae34-c442649f1fce","Type":"ContainerDied","Data":"f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470"}
Sep 09 21:10:14.574130 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:14.574101    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8_1dbf776a-2d18-4dd7-ae34-c442649f1fce/storage-initializer/0.log"
Sep 09 21:10:14.574497 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:14.574185    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8" event={"ID":"1dbf776a-2d18-4dd7-ae34-c442649f1fce","Type":"ContainerStarted","Data":"eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b"}
Sep 09 21:10:16.677770 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:16.677739    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"]
Sep 09 21:10:16.678204 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:16.677931    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerName="storage-initializer" containerID="cri-o://eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b" gracePeriod=30
Sep 09 21:10:18.329843 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.329823    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8_1dbf776a-2d18-4dd7-ae34-c442649f1fce/storage-initializer/1.log"
Sep 09 21:10:18.330234 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.330218    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8_1dbf776a-2d18-4dd7-ae34-c442649f1fce/storage-initializer/0.log"
Sep 09 21:10:18.332427 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.332413    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:18.397257 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397162    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbf776a-2d18-4dd7-ae34-c442649f1fce-proxy-tls\") pod \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") "
Sep 09 21:10:18.397257 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397204    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-rbac-proxy-sar-config\") pod \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") "
Sep 09 21:10:18.397257 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397232    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-cabundle-cert\") pod \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") "
Sep 09 21:10:18.397532 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397328    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bf8p\" (UniqueName: \"kubernetes.io/projected/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-api-access-4bf8p\") pod \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") "
Sep 09 21:10:18.397532 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397405    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kserve-provision-location\") pod \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\" (UID: \"1dbf776a-2d18-4dd7-ae34-c442649f1fce\") "
Sep 09 21:10:18.397654 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397569    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1dbf776a-2d18-4dd7-ae34-c442649f1fce" (UID: "1dbf776a-2d18-4dd7-ae34-c442649f1fce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:18.397654 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397591    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1dbf776a-2d18-4dd7-ae34-c442649f1fce" (UID: "1dbf776a-2d18-4dd7-ae34-c442649f1fce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:18.397774 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397672    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:18.397774 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397688    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf776a-2d18-4dd7-ae34-c442649f1fce-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:18.397774 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.397687    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dbf776a-2d18-4dd7-ae34-c442649f1fce" (UID: "1dbf776a-2d18-4dd7-ae34-c442649f1fce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:18.399487 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.399464    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-api-access-4bf8p" (OuterVolumeSpecName: "kube-api-access-4bf8p") pod "1dbf776a-2d18-4dd7-ae34-c442649f1fce" (UID: "1dbf776a-2d18-4dd7-ae34-c442649f1fce"). InnerVolumeSpecName "kube-api-access-4bf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:18.399602 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.399582    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbf776a-2d18-4dd7-ae34-c442649f1fce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dbf776a-2d18-4dd7-ae34-c442649f1fce" (UID: "1dbf776a-2d18-4dd7-ae34-c442649f1fce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:18.498320 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.498283    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dbf776a-2d18-4dd7-ae34-c442649f1fce-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:18.498320 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.498311    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bf8p\" (UniqueName: \"kubernetes.io/projected/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kube-api-access-4bf8p\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:18.498320 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.498322    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dbf776a-2d18-4dd7-ae34-c442649f1fce-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:10:18.590125 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.590098    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8_1dbf776a-2d18-4dd7-ae34-c442649f1fce/storage-initializer/1.log"
Sep 09 21:10:18.590500 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.590484    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8_1dbf776a-2d18-4dd7-ae34-c442649f1fce/storage-initializer/0.log"
Sep 09 21:10:18.590541 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.590518    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerID="eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b" exitCode=1
Sep 09 21:10:18.590630 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.590543    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8" event={"ID":"1dbf776a-2d18-4dd7-ae34-c442649f1fce","Type":"ContainerDied","Data":"eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b"}
Sep 09 21:10:18.590630 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.590601    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"
Sep 09 21:10:18.590630 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.590620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8" event={"ID":"1dbf776a-2d18-4dd7-ae34-c442649f1fce","Type":"ContainerDied","Data":"2618ca6027fdfe34ee18582f4c40e877eeee8a0cedf54a93c38ef93957ae3b01"}
Sep 09 21:10:18.590726 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.590645    2573 scope.go:117] "RemoveContainer" containerID="eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b"
Sep 09 21:10:18.599172 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.599156    2573 scope.go:117] "RemoveContainer" containerID="f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470"
Sep 09 21:10:18.606418 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.606403    2573 scope.go:117] "RemoveContainer" containerID="eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b"
Sep 09 21:10:18.606691 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:10:18.606669    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b\": container with ID starting with eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b not found: ID does not exist" containerID="eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b"
Sep 09 21:10:18.606763 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.606698    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b"} err="failed to get container status \"eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b\": rpc error: code = NotFound desc = could not find container \"eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b\": container with ID starting with eb64597fbcbb99e2c7ba5054e1e13bdd13a5d6dfd5e0c6d6d6d4588e01ec779b not found: ID does not exist"
Sep 09 21:10:18.606763 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.606713    2573 scope.go:117] "RemoveContainer" containerID="f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470"
Sep 09 21:10:18.606951 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:10:18.606933    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470\": container with ID starting with f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470 not found: ID does not exist" containerID="f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470"
Sep 09 21:10:18.606989 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.606958    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470"} err="failed to get container status \"f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470\": rpc error: code = NotFound desc = could not find container \"f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470\": container with ID starting with f75a99a857e88ce13ddf534aacd72d5f82452851dc24389963589f8ee9b95470 not found: ID does not exist"
Sep 09 21:10:18.630263 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.630238    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"]
Sep 09 21:10:18.633998 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:18.633977    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2aef6d-predictor-9587cb69d-v5gx8"]
Sep 09 21:10:20.286775 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:20.286745    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" path="/var/lib/kubelet/pods/1dbf776a-2d18-4dd7-ae34-c442649f1fce/volumes"
Sep 09 21:10:21.999098 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:21.999062    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"]
Sep 09 21:10:21.999467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:21.999345    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerName="storage-initializer"
Sep 09 21:10:21.999467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:21.999356    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerName="storage-initializer"
Sep 09 21:10:21.999467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:21.999428    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerName="storage-initializer"
Sep 09 21:10:21.999467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:21.999437    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerName="storage-initializer"
Sep 09 21:10:21.999646 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:21.999487    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerName="storage-initializer"
Sep 09 21:10:21.999646 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:21.999494    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbf776a-2d18-4dd7-ae34-c442649f1fce" containerName="storage-initializer"
Sep 09 21:10:22.004037 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.004020    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.006087 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.006063    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-d113f-predictor-serving-cert\""
Sep 09 21:10:22.006257 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.006178    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:10:22.006257 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.006190    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:10:22.006431 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.006418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:10:22.006486 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.006467    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:10:22.007043 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.007026    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-d113f-kube-rbac-proxy-sar-config\""
Sep 09 21:10:22.007101 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.007047    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:10:22.012973 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.012955    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"]
Sep 09 21:10:22.127368 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.127326    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-cabundle-cert\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.127368 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.127371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41323af8-d335-476c-b826-d585901ebc4d-kserve-provision-location\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.127619 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.127390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41323af8-d335-476c-b826-d585901ebc4d-proxy-tls\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.127619 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.127475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.127619 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.127528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56tfz\" (UniqueName: \"kubernetes.io/projected/41323af8-d335-476c-b826-d585901ebc4d-kube-api-access-56tfz\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.228396 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.228363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-cabundle-cert\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.228596 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.228403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41323af8-d335-476c-b826-d585901ebc4d-kserve-provision-location\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.228596 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.228422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41323af8-d335-476c-b826-d585901ebc4d-proxy-tls\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.228596 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.228471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.228596 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.228517    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56tfz\" (UniqueName: \"kubernetes.io/projected/41323af8-d335-476c-b826-d585901ebc4d-kube-api-access-56tfz\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.228865 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.228844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41323af8-d335-476c-b826-d585901ebc4d-kserve-provision-location\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.229089 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.229070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-cabundle-cert\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.229164 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.229147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.231032 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.231011    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41323af8-d335-476c-b826-d585901ebc4d-proxy-tls\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.236743 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.236722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56tfz\" (UniqueName: \"kubernetes.io/projected/41323af8-d335-476c-b826-d585901ebc4d-kube-api-access-56tfz\") pod \"raw-sklearn-d113f-predictor-6d97c45b8b-t46j2\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") " pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.313724 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.313704    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:22.587334 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.587296    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"]
Sep 09 21:10:22.589671 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:10:22.589644    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41323af8_d335_476c_b826_d585901ebc4d.slice/crio-3a8fa64f09de2cdd12df50646980a9267a8f44bd404b8d7e435efa5261f4ddb1 WatchSource:0}: Error finding container 3a8fa64f09de2cdd12df50646980a9267a8f44bd404b8d7e435efa5261f4ddb1: Status 404 returned error can't find the container with id 3a8fa64f09de2cdd12df50646980a9267a8f44bd404b8d7e435efa5261f4ddb1
Sep 09 21:10:22.607044 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:22.607015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerStarted","Data":"3a8fa64f09de2cdd12df50646980a9267a8f44bd404b8d7e435efa5261f4ddb1"}
Sep 09 21:10:23.612355 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:23.612319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerStarted","Data":"71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa"}
Sep 09 21:10:26.624730 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:26.624699    2573 generic.go:356] "Generic (PLEG): container finished" podID="41323af8-d335-476c-b826-d585901ebc4d" containerID="71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa" exitCode=0
Sep 09 21:10:26.625091 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:26.624757    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerDied","Data":"71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa"}
Sep 09 21:10:27.630686 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:27.630654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerStarted","Data":"7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8"}
Sep 09 21:10:27.631121 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:27.630714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerStarted","Data":"5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132"}
Sep 09 21:10:27.631121 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:27.630981    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:27.631121 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:27.631116    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:27.632524 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:27.632491    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:10:27.650598 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:27.650532    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podStartSLOduration=2.8703080720000003 podStartE2EDuration="6.650519478s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780211406s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:27.649065662 +0000 UTC m=+1099.859609438" watchObservedRunningTime="2026-09-09 21:10:27.650519478 +0000 UTC m=+1099.861063261"
Sep 09 21:10:28.634695 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:28.634658    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:10:33.640951 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:33.640917    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:10:33.641493 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:33.641393    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:10:43.641877 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:43.641837    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:10:53.641567 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:10:53.641512    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:11:03.641892 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:03.641851    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:11:13.642194 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:13.642149    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:11:23.641344 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:23.641303    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:11:33.641827 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:33.641782    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:11:38.284459 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:38.284425    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 21:11:48.288030 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:48.288002    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:11:52.026825 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:52.026792    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"]
Sep 09 21:11:52.027284 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:52.027051    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container" containerID="cri-o://5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132" gracePeriod=30
Sep 09 21:11:52.027284 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:52.027062    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kube-rbac-proxy" containerID="cri-o://7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8" gracePeriod=30
Sep 09 21:11:52.956509 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:52.956475    2573 generic.go:356] "Generic (PLEG): container finished" podID="41323af8-d335-476c-b826-d585901ebc4d" containerID="7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8" exitCode=2
Sep 09 21:11:52.956689 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:52.956565    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerDied","Data":"7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8"}
Sep 09 21:11:53.635805 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:53.635768    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 09 21:11:56.771403 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.771375    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:11:56.871747 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.871649    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-kube-rbac-proxy-sar-config\") pod \"41323af8-d335-476c-b826-d585901ebc4d\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") "
Sep 09 21:11:56.871747 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.871699    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-cabundle-cert\") pod \"41323af8-d335-476c-b826-d585901ebc4d\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") "
Sep 09 21:11:56.871747 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.871736    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41323af8-d335-476c-b826-d585901ebc4d-proxy-tls\") pod \"41323af8-d335-476c-b826-d585901ebc4d\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") "
Sep 09 21:11:56.872033 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.871781    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56tfz\" (UniqueName: \"kubernetes.io/projected/41323af8-d335-476c-b826-d585901ebc4d-kube-api-access-56tfz\") pod \"41323af8-d335-476c-b826-d585901ebc4d\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") "
Sep 09 21:11:56.872033 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.871815    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41323af8-d335-476c-b826-d585901ebc4d-kserve-provision-location\") pod \"41323af8-d335-476c-b826-d585901ebc4d\" (UID: \"41323af8-d335-476c-b826-d585901ebc4d\") "
Sep 09 21:11:56.872140 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.872054    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41323af8-d335-476c-b826-d585901ebc4d" (UID: "41323af8-d335-476c-b826-d585901ebc4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:11:56.872140 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.872131    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:11:56.872222 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.872141    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41323af8-d335-476c-b826-d585901ebc4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41323af8-d335-476c-b826-d585901ebc4d" (UID: "41323af8-d335-476c-b826-d585901ebc4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:56.872222 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.872145    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41323af8-d335-476c-b826-d585901ebc4d" (UID: "41323af8-d335-476c-b826-d585901ebc4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:11:56.874087 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.874067    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41323af8-d335-476c-b826-d585901ebc4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41323af8-d335-476c-b826-d585901ebc4d" (UID: "41323af8-d335-476c-b826-d585901ebc4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:11:56.874173 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.874128    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41323af8-d335-476c-b826-d585901ebc4d-kube-api-access-56tfz" (OuterVolumeSpecName: "kube-api-access-56tfz") pod "41323af8-d335-476c-b826-d585901ebc4d" (UID: "41323af8-d335-476c-b826-d585901ebc4d"). InnerVolumeSpecName "kube-api-access-56tfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:11:56.972507 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.972476    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41323af8-d335-476c-b826-d585901ebc4d-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:11:56.972507 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.972504    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41323af8-d335-476c-b826-d585901ebc4d-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:11:56.972717 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.972518    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41323af8-d335-476c-b826-d585901ebc4d-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:11:56.972717 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.972531    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56tfz\" (UniqueName: \"kubernetes.io/projected/41323af8-d335-476c-b826-d585901ebc4d-kube-api-access-56tfz\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:11:56.973359 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.973333    2573 generic.go:356] "Generic (PLEG): container finished" podID="41323af8-d335-476c-b826-d585901ebc4d" containerID="5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132" exitCode=0
Sep 09 21:11:56.973454 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.973392    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"
Sep 09 21:11:56.973454 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.973427    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerDied","Data":"5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132"}
Sep 09 21:11:56.973578 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.973504    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2" event={"ID":"41323af8-d335-476c-b826-d585901ebc4d","Type":"ContainerDied","Data":"3a8fa64f09de2cdd12df50646980a9267a8f44bd404b8d7e435efa5261f4ddb1"}
Sep 09 21:11:56.973578 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.973564    2573 scope.go:117] "RemoveContainer" containerID="7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8"
Sep 09 21:11:56.982104 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.982086    2573 scope.go:117] "RemoveContainer" containerID="5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132"
Sep 09 21:11:56.989258 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.989241    2573 scope.go:117] "RemoveContainer" containerID="71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa"
Sep 09 21:11:56.995103 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.995079    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"]
Sep 09 21:11:56.997021 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.997001    2573 scope.go:117] "RemoveContainer" containerID="7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8"
Sep 09 21:11:56.997370 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:11:56.997310    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8\": container with ID starting with 7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8 not found: ID does not exist" containerID="7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8"
Sep 09 21:11:56.997370 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.997355    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8"} err="failed to get container status \"7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8\": rpc error: code = NotFound desc = could not find container \"7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8\": container with ID starting with 7a78e9757e04df3c862a902b1c602ededf7fe91a891b76fd5398edeb8e4efef8 not found: ID does not exist"
Sep 09 21:11:56.997370 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.997381    2573 scope.go:117] "RemoveContainer" containerID="5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132"
Sep 09 21:11:56.997849 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:11:56.997818    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132\": container with ID starting with 5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132 not found: ID does not exist" containerID="5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132"
Sep 09 21:11:56.998006 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.997858    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132"} err="failed to get container status \"5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132\": rpc error: code = NotFound desc = could not find container \"5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132\": container with ID starting with 5ff54d16a20d942bff9bf6e930c50d16504c8acbe26a14aac4afcaab67623132 not found: ID does not exist"
Sep 09 21:11:56.998006 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.997882    2573 scope.go:117] "RemoveContainer" containerID="71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa"
Sep 09 21:11:56.998202 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:11:56.998178    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa\": container with ID starting with 71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa not found: ID does not exist" containerID="71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa"
Sep 09 21:11:56.998299 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.998210    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa"} err="failed to get container status \"71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa\": rpc error: code = NotFound desc = could not find container \"71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa\": container with ID starting with 71d1108ed4e4997fa3aab1fdc5251f851298ef933be690c76ce3ba1342b192aa not found: ID does not exist"
Sep 09 21:11:56.999700 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:56.999681    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-d113f-predictor-6d97c45b8b-t46j2"]
Sep 09 21:11:57.260910 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.260829    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"]
Sep 09 21:11:57.261130 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261117    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="storage-initializer"
Sep 09 21:11:57.261189 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261132    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="storage-initializer"
Sep 09 21:11:57.261189 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261141    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container"
Sep 09 21:11:57.261189 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261146    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container"
Sep 09 21:11:57.261189 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261151    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kube-rbac-proxy"
Sep 09 21:11:57.261189 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261158    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kube-rbac-proxy"
Sep 09 21:11:57.261337 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261219    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kube-rbac-proxy"
Sep 09 21:11:57.261337 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.261227    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41323af8-d335-476c-b826-d585901ebc4d" containerName="kserve-container"
Sep 09 21:11:57.265605 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.265585    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.267951 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.267931    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c1c50-predictor-serving-cert\""
Sep 09 21:11:57.267951 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.267955    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:11:57.268276 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.268248    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:11:57.268276 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.268266    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:11:57.268435 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.268376    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:11:57.268589 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.268547    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-c1c50-kube-rbac-proxy-sar-config\""
Sep 09 21:11:57.268792 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.268768    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:11:57.275785 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.275765    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"]
Sep 09 21:11:57.376240 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.376199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j467v\" (UniqueName: \"kubernetes.io/projected/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-api-access-j467v\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.376418 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.376238    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-cabundle-cert\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.376418 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.376381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.376418 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.376410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316cac1f-66b6-43e2-8d34-3da48a0e39f3-proxy-tls\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.376545 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.376441    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kserve-provision-location\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.477320 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.477284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kserve-provision-location\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.477508 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.477340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j467v\" (UniqueName: \"kubernetes.io/projected/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-api-access-j467v\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.477508 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.477360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-cabundle-cert\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.477508 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.477402    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.477508 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.477435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316cac1f-66b6-43e2-8d34-3da48a0e39f3-proxy-tls\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.477777 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.477751    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kserve-provision-location\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.478043 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.478014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-cabundle-cert\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.478147 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.478101    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.479950 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.479926    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316cac1f-66b6-43e2-8d34-3da48a0e39f3-proxy-tls\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.487080 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.487061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j467v\" (UniqueName: \"kubernetes.io/projected/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-api-access-j467v\") pod \"raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.575665 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.575615    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:11:57.847184 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.847059    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"]
Sep 09 21:11:57.849520 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:11:57.849486    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316cac1f_66b6_43e2_8d34_3da48a0e39f3.slice/crio-50405161a1599a87d1c8fffb210c285966969a222a8c2643683e3ac66f481af3 WatchSource:0}: Error finding container 50405161a1599a87d1c8fffb210c285966969a222a8c2643683e3ac66f481af3: Status 404 returned error can't find the container with id 50405161a1599a87d1c8fffb210c285966969a222a8c2643683e3ac66f481af3
Sep 09 21:11:57.978577 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.978515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerStarted","Data":"06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0"}
Sep 09 21:11:57.978770 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:57.978624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerStarted","Data":"50405161a1599a87d1c8fffb210c285966969a222a8c2643683e3ac66f481af3"}
Sep 09 21:11:58.289365 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:11:58.289284    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41323af8-d335-476c-b826-d585901ebc4d" path="/var/lib/kubelet/pods/41323af8-d335-476c-b826-d585901ebc4d/volumes"
Sep 09 21:12:01.995427 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:01.995338    2573 generic.go:356] "Generic (PLEG): container finished" podID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerID="06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0" exitCode=0
Sep 09 21:12:01.995427 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:01.995388    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerDied","Data":"06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0"}
Sep 09 21:12:03.001193 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:03.001155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerStarted","Data":"f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e"}
Sep 09 21:12:03.001608 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:03.001222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerStarted","Data":"8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc"}
Sep 09 21:12:03.001608 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:03.001508    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:12:03.001608 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:03.001589    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:12:03.003036 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:03.003010    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:12:03.020589 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:03.020509    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podStartSLOduration=2.1970257970000002 podStartE2EDuration="6.020490572s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.823464775s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:03.01993831 +0000 UTC m=+1195.230482104" watchObservedRunningTime="2026-09-09 21:12:03.020490572 +0000 UTC m=+1195.231034374"
Sep 09 21:12:04.005267 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:04.005230    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:12:08.294154 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:08.294128    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:12:08.297024 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:08.297002    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:12:09.010938 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:09.010910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:12:09.011408 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:09.011382    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:12:19.012297 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:19.012259    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:12:29.011334 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:29.011292    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:12:39.011388 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:39.011346    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:12:49.012196 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:49.012151    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:12:59.011544 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:12:59.011493    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:13:09.012191 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:09.012148    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:13:13.283094 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:13.283047    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 21:13:19.227199 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.227159    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"]
Sep 09 21:13:19.230572 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.230532    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.233665 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.233643    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 21:13:19.233768 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.233684    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 21:13:19.251326 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.251297    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"]
Sep 09 21:13:19.403291 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.403252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.403467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.403293    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.403467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.403323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.403467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.403355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vk4r\" (UniqueName: \"kubernetes.io/projected/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-api-access-9vk4r\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.403467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.403431    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.504825 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.504743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.504825 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.504782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.504825 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.504811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vk4r\" (UniqueName: \"kubernetes.io/projected/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-api-access-9vk4r\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.505065 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.504874    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.505065 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.504930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.505329 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.505305    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.505480 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.505458    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.505576 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.505486    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.507407 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.507387    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.513241 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.513219    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vk4r\" (UniqueName: \"kubernetes.io/projected/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-api-access-9vk4r\") pod \"isvc-canary-create-predictor-854596548c-vbp28\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.542277 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.542243    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:19.816353 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:19.816319    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"]
Sep 09 21:13:19.819333 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:13:19.819297    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1bc169_1dd7_444f_8c6d_8de9cde866b9.slice/crio-c977dac391f1fce54d01747825b717ec1cce87e4ca1107647b2e3e9140e11df6 WatchSource:0}: Error finding container c977dac391f1fce54d01747825b717ec1cce87e4ca1107647b2e3e9140e11df6: Status 404 returned error can't find the container with id c977dac391f1fce54d01747825b717ec1cce87e4ca1107647b2e3e9140e11df6
Sep 09 21:13:20.295621 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:20.295583    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerStarted","Data":"3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667"}
Sep 09 21:13:20.296073 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:20.295655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerStarted","Data":"c977dac391f1fce54d01747825b717ec1cce87e4ca1107647b2e3e9140e11df6"}
Sep 09 21:13:23.283993 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:23.283963    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:13:24.311328 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:24.311287    2573 generic.go:356] "Generic (PLEG): container finished" podID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerID="3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667" exitCode=0
Sep 09 21:13:24.311753 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:24.311362    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerDied","Data":"3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667"}
Sep 09 21:13:25.317412 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:25.317376    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerStarted","Data":"8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b"}
Sep 09 21:13:25.317804 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:25.317461    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerStarted","Data":"bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd"}
Sep 09 21:13:25.317804 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:25.317664    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:25.337819 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:25.337774    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podStartSLOduration=2.488422961 podStartE2EDuration="6.337759534s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.849336573s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:25.336187884 +0000 UTC m=+1277.546731709" watchObservedRunningTime="2026-09-09 21:13:25.337759534 +0000 UTC m=+1277.548303317"
Sep 09 21:13:26.321506 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:26.321468    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:26.322789 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:26.322762    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:13:27.308902 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:27.308871    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"]
Sep 09 21:13:27.309164 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:27.309124    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container" containerID="cri-o://8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc" gracePeriod=30
Sep 09 21:13:27.309236 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:27.309194    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kube-rbac-proxy" containerID="cri-o://f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e" gracePeriod=30
Sep 09 21:13:27.324986 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:27.324958    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:13:28.330501 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:28.330465    2573 generic.go:356] "Generic (PLEG): container finished" podID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerID="f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e" exitCode=2
Sep 09 21:13:28.330886 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:28.330536    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerDied","Data":"f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e"}
Sep 09 21:13:29.006000 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:29.005961    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 09 21:13:32.049754 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.049725    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:13:32.098840 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.098806    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-rbac-proxy-sar-config\") pod \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") "
Sep 09 21:13:32.099003 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.098864    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j467v\" (UniqueName: \"kubernetes.io/projected/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-api-access-j467v\") pod \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") "
Sep 09 21:13:32.099003 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.098890    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316cac1f-66b6-43e2-8d34-3da48a0e39f3-proxy-tls\") pod \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") "
Sep 09 21:13:32.099003 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.098923    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kserve-provision-location\") pod \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") "
Sep 09 21:13:32.099003 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.098969    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-cabundle-cert\") pod \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\" (UID: \"316cac1f-66b6-43e2-8d34-3da48a0e39f3\") "
Sep 09 21:13:32.099305 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.099280    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "316cac1f-66b6-43e2-8d34-3da48a0e39f3" (UID: "316cac1f-66b6-43e2-8d34-3da48a0e39f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:32.099370 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.099277    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "316cac1f-66b6-43e2-8d34-3da48a0e39f3" (UID: "316cac1f-66b6-43e2-8d34-3da48a0e39f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:32.099370 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.099355    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "316cac1f-66b6-43e2-8d34-3da48a0e39f3" (UID: "316cac1f-66b6-43e2-8d34-3da48a0e39f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:13:32.101226 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.101205    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316cac1f-66b6-43e2-8d34-3da48a0e39f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "316cac1f-66b6-43e2-8d34-3da48a0e39f3" (UID: "316cac1f-66b6-43e2-8d34-3da48a0e39f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:32.101303 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.101234    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-api-access-j467v" (OuterVolumeSpecName: "kube-api-access-j467v") pod "316cac1f-66b6-43e2-8d34-3da48a0e39f3" (UID: "316cac1f-66b6-43e2-8d34-3da48a0e39f3"). InnerVolumeSpecName "kube-api-access-j467v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:32.199470 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.199390    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j467v\" (UniqueName: \"kubernetes.io/projected/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-api-access-j467v\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:13:32.199470 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.199419    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316cac1f-66b6-43e2-8d34-3da48a0e39f3-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:13:32.199470 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.199429    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:13:32.199470 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.199437    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:13:32.199470 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.199452    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316cac1f-66b6-43e2-8d34-3da48a0e39f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:13:32.330202 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.330175    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:13:32.330751 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.330728    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:13:32.347073 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.347040    2573 generic.go:356] "Generic (PLEG): container finished" podID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerID="8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc" exitCode=0
Sep 09 21:13:32.347232 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.347099    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"
Sep 09 21:13:32.347232 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.347123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerDied","Data":"8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc"}
Sep 09 21:13:32.347232 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.347190    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77" event={"ID":"316cac1f-66b6-43e2-8d34-3da48a0e39f3","Type":"ContainerDied","Data":"50405161a1599a87d1c8fffb210c285966969a222a8c2643683e3ac66f481af3"}
Sep 09 21:13:32.347393 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.347226    2573 scope.go:117] "RemoveContainer" containerID="f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e"
Sep 09 21:13:32.356030 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.356007    2573 scope.go:117] "RemoveContainer" containerID="8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc"
Sep 09 21:13:32.363646 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.363631    2573 scope.go:117] "RemoveContainer" containerID="06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0"
Sep 09 21:13:32.366348 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.366325    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"]
Sep 09 21:13:32.371739 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.371675    2573 scope.go:117] "RemoveContainer" containerID="f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e"
Sep 09 21:13:32.372027 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:13:32.371999    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e\": container with ID starting with f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e not found: ID does not exist" containerID="f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e"
Sep 09 21:13:32.372131 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.372033    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e"} err="failed to get container status \"f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e\": rpc error: code = NotFound desc = could not find container \"f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e\": container with ID starting with f6f6edd30261bbca4bcf794404c2c0c9db9f69560b88981af1662f0c9a6f330e not found: ID does not exist"
Sep 09 21:13:32.372131 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.372053    2573 scope.go:117] "RemoveContainer" containerID="8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc"
Sep 09 21:13:32.372307 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:13:32.372292    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc\": container with ID starting with 8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc not found: ID does not exist" containerID="8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc"
Sep 09 21:13:32.372344 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.372311    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc"} err="failed to get container status \"8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc\": rpc error: code = NotFound desc = could not find container \"8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc\": container with ID starting with 8f21b5567f45e2de12f5d65ef4bfa10343fd0d6aa88b74f38d88aa0b850b8bdc not found: ID does not exist"
Sep 09 21:13:32.372344 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.372324    2573 scope.go:117] "RemoveContainer" containerID="06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0"
Sep 09 21:13:32.372652 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:13:32.372630    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0\": container with ID starting with 06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0 not found: ID does not exist" containerID="06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0"
Sep 09 21:13:32.372718 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.372662    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0"} err="failed to get container status \"06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0\": rpc error: code = NotFound desc = could not find container \"06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0\": container with ID starting with 06ae11a1d34438c705d64afcc76ba32f489b3f7d85febdd206ac6bd2956de9e0 not found: ID does not exist"
Sep 09 21:13:32.373261 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:32.373244    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-c1c50-predictor-6f8767b686-bxz77"]
Sep 09 21:13:34.286899 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:34.286868    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" path="/var/lib/kubelet/pods/316cac1f-66b6-43e2-8d34-3da48a0e39f3/volumes"
Sep 09 21:13:42.331070 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:42.331029    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:13:52.331580 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:13:52.331513    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:14:02.334060 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:02.334013    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:14:12.331025 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:12.330982    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:14:22.330972 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:22.330935    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:14:32.330776 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:32.330690    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 21:14:41.432894 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:41.432842    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"]
Sep 09 21:14:41.433323 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:41.433071    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container" containerID="cri-o://bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd" gracePeriod=30
Sep 09 21:14:41.433323 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:41.433147    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kube-rbac-proxy" containerID="cri-o://8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b" gracePeriod=30
Sep 09 21:14:41.433891 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:41.433853    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:14:41.613705 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:41.613665    2573 generic.go:356] "Generic (PLEG): container finished" podID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerID="8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b" exitCode=2
Sep 09 21:14:41.613867 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:41.613715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerDied","Data":"8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b"}
Sep 09 21:14:42.325281 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:42.325240    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 09 21:14:46.166486 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.166462    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:14:46.358483 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.358449    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-proxy-tls\") pod \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") "
Sep 09 21:14:46.358674 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.358495    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-cabundle-cert\") pod \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") "
Sep 09 21:14:46.358674 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.358536    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kserve-provision-location\") pod \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") "
Sep 09 21:14:46.358674 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.358658    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vk4r\" (UniqueName: \"kubernetes.io/projected/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-api-access-9vk4r\") pod \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") "
Sep 09 21:14:46.358818 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.358735    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-rbac-proxy-sar-config\") pod \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\" (UID: \"7f1bc169-1dd7-444f-8c6d-8de9cde866b9\") "
Sep 09 21:14:46.358910 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.358880    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f1bc169-1dd7-444f-8c6d-8de9cde866b9" (UID: "7f1bc169-1dd7-444f-8c6d-8de9cde866b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:46.358910 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.358901    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f1bc169-1dd7-444f-8c6d-8de9cde866b9" (UID: "7f1bc169-1dd7-444f-8c6d-8de9cde866b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:46.359103 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.359045    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.359103 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.359058    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.359181 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.359112    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f1bc169-1dd7-444f-8c6d-8de9cde866b9" (UID: "7f1bc169-1dd7-444f-8c6d-8de9cde866b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:46.360825 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.360803    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-api-access-9vk4r" (OuterVolumeSpecName: "kube-api-access-9vk4r") pod "7f1bc169-1dd7-444f-8c6d-8de9cde866b9" (UID: "7f1bc169-1dd7-444f-8c6d-8de9cde866b9"). InnerVolumeSpecName "kube-api-access-9vk4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:46.360911 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.360826    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f1bc169-1dd7-444f-8c6d-8de9cde866b9" (UID: "7f1bc169-1dd7-444f-8c6d-8de9cde866b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:46.459581 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.459518    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.459581 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.459546    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vk4r\" (UniqueName: \"kubernetes.io/projected/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-api-access-9vk4r\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.459581 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.459588    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f1bc169-1dd7-444f-8c6d-8de9cde866b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:14:46.634581 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.634480    2573 generic.go:356] "Generic (PLEG): container finished" podID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerID="bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd" exitCode=0
Sep 09 21:14:46.634581 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.634532    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"
Sep 09 21:14:46.634755 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.634533    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerDied","Data":"bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd"}
Sep 09 21:14:46.634755 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.634670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28" event={"ID":"7f1bc169-1dd7-444f-8c6d-8de9cde866b9","Type":"ContainerDied","Data":"c977dac391f1fce54d01747825b717ec1cce87e4ca1107647b2e3e9140e11df6"}
Sep 09 21:14:46.634755 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.634711    2573 scope.go:117] "RemoveContainer" containerID="8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b"
Sep 09 21:14:46.643795 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.643779    2573 scope.go:117] "RemoveContainer" containerID="bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd"
Sep 09 21:14:46.650823 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.650803    2573 scope.go:117] "RemoveContainer" containerID="3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667"
Sep 09 21:14:46.658853 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.657725    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"]
Sep 09 21:14:46.662100 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.662069    2573 scope.go:117] "RemoveContainer" containerID="8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b"
Sep 09 21:14:46.662667 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:14:46.662644    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b\": container with ID starting with 8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b not found: ID does not exist" containerID="8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b"
Sep 09 21:14:46.662748 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.662681    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b"} err="failed to get container status \"8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b\": rpc error: code = NotFound desc = could not find container \"8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b\": container with ID starting with 8306dc68ba018af835b51019d729764251cf039781314c2cb4a5e4f318018c6b not found: ID does not exist"
Sep 09 21:14:46.662748 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.662718    2573 scope.go:117] "RemoveContainer" containerID="bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd"
Sep 09 21:14:46.663004 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:14:46.662986    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd\": container with ID starting with bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd not found: ID does not exist" containerID="bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd"
Sep 09 21:14:46.663062 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.663009    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd"} err="failed to get container status \"bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd\": rpc error: code = NotFound desc = could not find container \"bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd\": container with ID starting with bdb8d49305db1126de17ca075cd7fe750da2936740a38ef37392ec33b8579afd not found: ID does not exist"
Sep 09 21:14:46.663062 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.663025    2573 scope.go:117] "RemoveContainer" containerID="3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667"
Sep 09 21:14:46.663299 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:14:46.663283    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667\": container with ID starting with 3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667 not found: ID does not exist" containerID="3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667"
Sep 09 21:14:46.663353 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.663303    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667"} err="failed to get container status \"3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667\": rpc error: code = NotFound desc = could not find container \"3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667\": container with ID starting with 3cdbe7f08186b55ae404dedcbbae072ea3d785793f6480b3a3a90f86c6c6e667 not found: ID does not exist"
Sep 09 21:14:46.664062 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:46.664042    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-vbp28"]
Sep 09 21:14:48.287594 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:48.287538    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" path="/var/lib/kubelet/pods/7f1bc169-1dd7-444f-8c6d-8de9cde866b9/volumes"
Sep 09 21:14:54.393691 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.393650    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"]
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.393967    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.393979    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.393986    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kube-rbac-proxy"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.393991    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kube-rbac-proxy"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394000    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394005    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394013    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kube-rbac-proxy"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394018    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kube-rbac-proxy"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394031    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="storage-initializer"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394039    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="storage-initializer"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394050    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="storage-initializer"
Sep 09 21:14:54.394075 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394054    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="storage-initializer"
Sep 09 21:14:54.394427 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394098    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kube-rbac-proxy"
Sep 09 21:14:54.394427 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394110    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kserve-container"
Sep 09 21:14:54.394427 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394121    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="316cac1f-66b6-43e2-8d34-3da48a0e39f3" containerName="kserve-container"
Sep 09 21:14:54.394427 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.394127    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f1bc169-1dd7-444f-8c6d-8de9cde866b9" containerName="kube-rbac-proxy"
Sep 09 21:14:54.398597 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.398579    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.401050 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.401029    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:14:54.401276 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.401262    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:14:54.401484 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.401472    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:14:54.402007 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.401990    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:14:54.402084 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.402004    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 21:14:54.402084 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.402006    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 21:14:54.402188 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.402010    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:14:54.413579 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.413538    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"]
Sep 09 21:14:54.423692 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.423666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.423798 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.423704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30dd331-0bbc-40ee-9371-6399dd27a808-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.423798 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.423731    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.423903 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.423823    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.423903 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.423849    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96nhh\" (UniqueName: \"kubernetes.io/projected/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-api-access-96nhh\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.524548 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.524506    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30dd331-0bbc-40ee-9371-6399dd27a808-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.524751 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.524585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.524751 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.524640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.524890 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.524761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96nhh\" (UniqueName: \"kubernetes.io/projected/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-api-access-96nhh\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.524890 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.524833    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.524988 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:14:54.524948    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 21:14:54.524988 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.524971    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30dd331-0bbc-40ee-9371-6399dd27a808-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.525086 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:14:54.525014    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls podName:b30dd331-0bbc-40ee-9371-6399dd27a808 nodeName:}" failed. No retries permitted until 2026-09-09 21:14:55.024998868 +0000 UTC m=+1367.235542631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls") pod "isvc-canary-promote-predictor-65ff4fc5f5-pffdr" (UID: "b30dd331-0bbc-40ee-9371-6399dd27a808") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 21:14:54.525242 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.525224    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.525301 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.525278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:54.535372 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:54.535343    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96nhh\" (UniqueName: \"kubernetes.io/projected/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-api-access-96nhh\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:55.028307 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:55.028268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:55.030852 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:55.030832    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-pffdr\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:55.308702 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:55.308664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:14:55.580255 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:55.580229    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"]
Sep 09 21:14:55.584107 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:14:55.584082    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30dd331_0bbc_40ee_9371_6399dd27a808.slice/crio-d960f9c76042d9065e030707682af7e9e34aea75044349ff2e51159b4110b9e3 WatchSource:0}: Error finding container d960f9c76042d9065e030707682af7e9e34aea75044349ff2e51159b4110b9e3: Status 404 returned error can't find the container with id d960f9c76042d9065e030707682af7e9e34aea75044349ff2e51159b4110b9e3
Sep 09 21:14:55.669124 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:55.669088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerStarted","Data":"5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f"}
Sep 09 21:14:55.669295 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:55.669158    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerStarted","Data":"d960f9c76042d9065e030707682af7e9e34aea75044349ff2e51159b4110b9e3"}
Sep 09 21:14:59.686368 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:59.686287    2573 generic.go:356] "Generic (PLEG): container finished" podID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerID="5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f" exitCode=0
Sep 09 21:14:59.686730 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:14:59.686363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerDied","Data":"5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f"}
Sep 09 21:15:00.692645 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:00.692611    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerStarted","Data":"309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7"}
Sep 09 21:15:00.693082 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:00.692671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerStarted","Data":"55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121"}
Sep 09 21:15:00.693082 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:00.692940    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:15:00.693189 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:00.693087    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:15:00.694472 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:00.694445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:15:00.714437 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:00.714388    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podStartSLOduration=3.055344093 podStartE2EDuration="6.714373712s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659029619s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:00.712602686 +0000 UTC m=+1372.923146469" watchObservedRunningTime="2026-09-09 21:15:00.714373712 +0000 UTC m=+1372.924917574"
Sep 09 21:15:01.696601 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:01.696541    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:15:06.701453 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:06.701422    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:15:06.702044 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:06.702018    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:15:16.701902 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:16.701855    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:15:26.702518 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:26.702479    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:15:36.701989 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:36.701941    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:15:46.702855 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:46.702818    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:15:56.702304 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:15:56.702266    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:16:06.702521 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:06.702434    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 21:16:15.421863 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:15.421820    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"]
Sep 09 21:16:15.422356 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:15.422155    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container" containerID="cri-o://55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121" gracePeriod=30
Sep 09 21:16:15.422356 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:15.422155    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kube-rbac-proxy" containerID="cri-o://309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7" gracePeriod=30
Sep 09 21:16:15.423281 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:15.423258    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:16:15.976068 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:15.976038    2573 generic.go:356] "Generic (PLEG): container finished" podID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerID="309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7" exitCode=2
Sep 09 21:16:15.976232 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:15.976082    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerDied","Data":"309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7"}
Sep 09 21:16:16.696988 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:16.696945    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 09 21:16:20.258074 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.258048    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:16:20.272391 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272371    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-cabundle-cert\") pod \"b30dd331-0bbc-40ee-9371-6399dd27a808\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") "
Sep 09 21:16:20.272465 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272402    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30dd331-0bbc-40ee-9371-6399dd27a808-kserve-provision-location\") pod \"b30dd331-0bbc-40ee-9371-6399dd27a808\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") "
Sep 09 21:16:20.272465 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272434    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96nhh\" (UniqueName: \"kubernetes.io/projected/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-api-access-96nhh\") pod \"b30dd331-0bbc-40ee-9371-6399dd27a808\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") "
Sep 09 21:16:20.272600 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272576    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls\") pod \"b30dd331-0bbc-40ee-9371-6399dd27a808\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") "
Sep 09 21:16:20.272671 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272654    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-rbac-proxy-sar-config\") pod \"b30dd331-0bbc-40ee-9371-6399dd27a808\" (UID: \"b30dd331-0bbc-40ee-9371-6399dd27a808\") "
Sep 09 21:16:20.272712 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272690    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30dd331-0bbc-40ee-9371-6399dd27a808-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b30dd331-0bbc-40ee-9371-6399dd27a808" (UID: "b30dd331-0bbc-40ee-9371-6399dd27a808"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:20.272768 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272750    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b30dd331-0bbc-40ee-9371-6399dd27a808" (UID: "b30dd331-0bbc-40ee-9371-6399dd27a808"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:20.272949 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272935    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.273015 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272950    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30dd331-0bbc-40ee-9371-6399dd27a808-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.273015 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.272982    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b30dd331-0bbc-40ee-9371-6399dd27a808" (UID: "b30dd331-0bbc-40ee-9371-6399dd27a808"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:20.274686 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.274659    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b30dd331-0bbc-40ee-9371-6399dd27a808" (UID: "b30dd331-0bbc-40ee-9371-6399dd27a808"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:20.274686 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.274669    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-api-access-96nhh" (OuterVolumeSpecName: "kube-api-access-96nhh") pod "b30dd331-0bbc-40ee-9371-6399dd27a808" (UID: "b30dd331-0bbc-40ee-9371-6399dd27a808"). InnerVolumeSpecName "kube-api-access-96nhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:20.373590 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.373485    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96nhh\" (UniqueName: \"kubernetes.io/projected/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-api-access-96nhh\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.373590 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.373516    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b30dd331-0bbc-40ee-9371-6399dd27a808-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.373590 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.373526    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b30dd331-0bbc-40ee-9371-6399dd27a808-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.997623 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.997590    2573 generic.go:356] "Generic (PLEG): container finished" podID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerID="55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121" exitCode=0
Sep 09 21:16:20.997836 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.997638    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerDied","Data":"55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121"}
Sep 09 21:16:20.997836 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.997646    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"
Sep 09 21:16:20.997836 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.997702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr" event={"ID":"b30dd331-0bbc-40ee-9371-6399dd27a808","Type":"ContainerDied","Data":"d960f9c76042d9065e030707682af7e9e34aea75044349ff2e51159b4110b9e3"}
Sep 09 21:16:20.997836 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:20.997736    2573 scope.go:117] "RemoveContainer" containerID="309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7"
Sep 09 21:16:21.006017 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.005996    2573 scope.go:117] "RemoveContainer" containerID="55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121"
Sep 09 21:16:21.013316 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.013296    2573 scope.go:117] "RemoveContainer" containerID="5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f"
Sep 09 21:16:21.018767 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.018745    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"]
Sep 09 21:16:21.020913 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.020891    2573 scope.go:117] "RemoveContainer" containerID="309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7"
Sep 09 21:16:21.021185 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:16:21.021168    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7\": container with ID starting with 309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7 not found: ID does not exist" containerID="309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7"
Sep 09 21:16:21.021237 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.021194    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7"} err="failed to get container status \"309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7\": rpc error: code = NotFound desc = could not find container \"309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7\": container with ID starting with 309fdd09b96311bb44fe247a3413244d90de201577f495d5f7383f2493f5cff7 not found: ID does not exist"
Sep 09 21:16:21.021237 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.021211    2573 scope.go:117] "RemoveContainer" containerID="55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121"
Sep 09 21:16:21.021444 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:16:21.021427    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121\": container with ID starting with 55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121 not found: ID does not exist" containerID="55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121"
Sep 09 21:16:21.021484 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.021457    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121"} err="failed to get container status \"55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121\": rpc error: code = NotFound desc = could not find container \"55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121\": container with ID starting with 55fe4e709da0aea254309c03eaa28a5a7264c8d1f66984f81288d574a9d9c121 not found: ID does not exist"
Sep 09 21:16:21.021484 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.021474    2573 scope.go:117] "RemoveContainer" containerID="5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f"
Sep 09 21:16:21.021733 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:16:21.021713    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f\": container with ID starting with 5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f not found: ID does not exist" containerID="5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f"
Sep 09 21:16:21.021778 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.021736    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f"} err="failed to get container status \"5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f\": rpc error: code = NotFound desc = could not find container \"5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f\": container with ID starting with 5bce4c2b72b24eb9bca07bb2bd7cbd1f6ba27d1c51769354d68eb00149bedf1f not found: ID does not exist"
Sep 09 21:16:21.023966 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:21.023945    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-pffdr"]
Sep 09 21:16:22.287461 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:22.287428    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" path="/var/lib/kubelet/pods/b30dd331-0bbc-40ee-9371-6399dd27a808/volumes"
Sep 09 21:16:24.566268 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566234    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"]
Sep 09 21:16:24.566668 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566527    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kube-rbac-proxy"
Sep 09 21:16:24.566668 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566539    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kube-rbac-proxy"
Sep 09 21:16:24.566668 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566578    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container"
Sep 09 21:16:24.566668 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566585    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container"
Sep 09 21:16:24.566668 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566593    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="storage-initializer"
Sep 09 21:16:24.566668 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566598    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="storage-initializer"
Sep 09 21:16:24.566861 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566673    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kube-rbac-proxy"
Sep 09 21:16:24.566861 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.566683    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b30dd331-0bbc-40ee-9371-6399dd27a808" containerName="kserve-container"
Sep 09 21:16:24.571367 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.571350    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.573884 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.573862    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:16:24.574004 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.573941    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:16:24.574004 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.573953    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:16:24.574004 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.573941    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 21:16:24.574671 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.574645    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:16:24.574759 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.574723    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:16:24.574759 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.574729    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 21:16:24.583300 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.583277    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"]
Sep 09 21:16:24.606294 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.606261    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d929\" (UniqueName: \"kubernetes.io/projected/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-api-access-8d929\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.606428 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.606298    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.606428 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.606334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.606428 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.606372    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.606527 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.606433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.707680 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.707642    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.707680 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.707681    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.707959 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.707710    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.707959 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.707752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d929\" (UniqueName: \"kubernetes.io/projected/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-api-access-8d929\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.707959 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.707776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.708136 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.708108    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.708367 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.708349    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.708412 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.708380    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.710438 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.710417    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.716735 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.716710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d929\" (UniqueName: \"kubernetes.io/projected/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-api-access-8d929\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-rbzv2\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:24.881523 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:24.881434    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:25.152808 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:25.152735    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"]
Sep 09 21:16:25.156032 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:16:25.156001    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fe733a_eb68_42c6_a9f4_38bff12e50c6.slice/crio-08d8c891602d62e2cc9ad8b1363e544ba028812a8658fdecb57a9a3ca23456e5 WatchSource:0}: Error finding container 08d8c891602d62e2cc9ad8b1363e544ba028812a8658fdecb57a9a3ca23456e5: Status 404 returned error can't find the container with id 08d8c891602d62e2cc9ad8b1363e544ba028812a8658fdecb57a9a3ca23456e5
Sep 09 21:16:26.019126 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:26.019086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerStarted","Data":"6b2899cf034aae8089bf067cf27ca4106d8e56e5a19068b244b77621ee4b9072"}
Sep 09 21:16:26.019505 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:26.019141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerStarted","Data":"08d8c891602d62e2cc9ad8b1363e544ba028812a8658fdecb57a9a3ca23456e5"}
Sep 09 21:16:29.033076 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:29.033044    2573 generic.go:356] "Generic (PLEG): container finished" podID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerID="6b2899cf034aae8089bf067cf27ca4106d8e56e5a19068b244b77621ee4b9072" exitCode=0
Sep 09 21:16:29.033480 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:29.033081    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerDied","Data":"6b2899cf034aae8089bf067cf27ca4106d8e56e5a19068b244b77621ee4b9072"}
Sep 09 21:16:30.038789 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:30.038752    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerStarted","Data":"08b3c65f1a6262bcaac8ecf2ee27e16ada217bc24f839ec7fbfedd128359b536"}
Sep 09 21:16:30.039212 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:30.038816    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerStarted","Data":"c94b9a5c392af22b30c9ee4ca94dc783b9e4ff94402bfebed0160969227a35fa"}
Sep 09 21:16:30.039212 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:30.039076    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:30.039303 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:30.039214    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:30.040487 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:30.040462    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:16:30.060842 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:30.060797    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podStartSLOduration=2.382216792 podStartE2EDuration="6.060780736s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.678563944s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:30.058828053 +0000 UTC m=+1462.269371837" watchObservedRunningTime="2026-09-09 21:16:30.060780736 +0000 UTC m=+1462.271324520"
Sep 09 21:16:31.042974 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:31.042929    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:16:36.047280 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:36.047247    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:16:36.047928 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:36.047894    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:16:46.048647 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:46.048602    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:16:56.047914 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:16:56.047873    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:17:06.048039 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:06.047998    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:17:08.319924 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:08.319893    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:17:08.323382 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:08.323355    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbk97_4cfab0e2-e12b-4fed-b42a-00bdf21fd385/ovn-acl-logging/0.log"
Sep 09 21:17:16.048442 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:16.048397    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:17:26.048473 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:26.048432    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:17:36.048276 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:36.048229    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:17:46.048757 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:46.048728    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:17:54.701011 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:54.700980    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"]
Sep 09 21:17:54.701442 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:54.701329    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" containerID="cri-o://c94b9a5c392af22b30c9ee4ca94dc783b9e4ff94402bfebed0160969227a35fa" gracePeriod=30
Sep 09 21:17:54.701442 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:54.701339    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kube-rbac-proxy" containerID="cri-o://08b3c65f1a6262bcaac8ecf2ee27e16ada217bc24f839ec7fbfedd128359b536" gracePeriod=30
Sep 09 21:17:55.355651 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:55.355616    2573 generic.go:356] "Generic (PLEG): container finished" podID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerID="08b3c65f1a6262bcaac8ecf2ee27e16ada217bc24f839ec7fbfedd128359b536" exitCode=2
Sep 09 21:17:55.355828 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:55.355684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerDied","Data":"08b3c65f1a6262bcaac8ecf2ee27e16ada217bc24f839ec7fbfedd128359b536"}
Sep 09 21:17:56.043257 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:56.043213    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 09 21:17:56.048763 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:56.048726    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 21:17:59.374524 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.374495    2573 generic.go:356] "Generic (PLEG): container finished" podID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerID="c94b9a5c392af22b30c9ee4ca94dc783b9e4ff94402bfebed0160969227a35fa" exitCode=0
Sep 09 21:17:59.374897 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.374581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerDied","Data":"c94b9a5c392af22b30c9ee4ca94dc783b9e4ff94402bfebed0160969227a35fa"}
Sep 09 21:17:59.438466 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.438441    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:17:59.548435 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548404    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kserve-provision-location\") pod \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") "
Sep 09 21:17:59.548645 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548478    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-proxy-tls\") pod \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") "
Sep 09 21:17:59.548645 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548505    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d929\" (UniqueName: \"kubernetes.io/projected/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-api-access-8d929\") pod \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") "
Sep 09 21:17:59.548645 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548528    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-cabundle-cert\") pod \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") "
Sep 09 21:17:59.548645 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548545    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-rbac-proxy-sar-config\") pod \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\" (UID: \"f9fe733a-eb68-42c6-a9f4-38bff12e50c6\") "
Sep 09 21:17:59.548918 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548833    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9fe733a-eb68-42c6-a9f4-38bff12e50c6" (UID: "f9fe733a-eb68-42c6-a9f4-38bff12e50c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:17:59.549048 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548933    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9fe733a-eb68-42c6-a9f4-38bff12e50c6" (UID: "f9fe733a-eb68-42c6-a9f4-38bff12e50c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:17:59.549048 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.548971    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9fe733a-eb68-42c6-a9f4-38bff12e50c6" (UID: "f9fe733a-eb68-42c6-a9f4-38bff12e50c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:17:59.550801 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.550780    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9fe733a-eb68-42c6-a9f4-38bff12e50c6" (UID: "f9fe733a-eb68-42c6-a9f4-38bff12e50c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:17:59.550912 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.550849    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-api-access-8d929" (OuterVolumeSpecName: "kube-api-access-8d929") pod "f9fe733a-eb68-42c6-a9f4-38bff12e50c6" (UID: "f9fe733a-eb68-42c6-a9f4-38bff12e50c6"). InnerVolumeSpecName "kube-api-access-8d929". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:17:59.650007 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.649972    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.650007 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.650003    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d929\" (UniqueName: \"kubernetes.io/projected/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-api-access-8d929\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.650007 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.650012    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-cabundle-cert\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.650236 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.650022    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:17:59.650236 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:17:59.650032    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9fe733a-eb68-42c6-a9f4-38bff12e50c6-kserve-provision-location\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:18:00.380265 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:00.380232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2" event={"ID":"f9fe733a-eb68-42c6-a9f4-38bff12e50c6","Type":"ContainerDied","Data":"08d8c891602d62e2cc9ad8b1363e544ba028812a8658fdecb57a9a3ca23456e5"}
Sep 09 21:18:00.380700 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:00.380281    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"
Sep 09 21:18:00.380700 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:00.380310    2573 scope.go:117] "RemoveContainer" containerID="08b3c65f1a6262bcaac8ecf2ee27e16ada217bc24f839ec7fbfedd128359b536"
Sep 09 21:18:00.388300 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:00.388281    2573 scope.go:117] "RemoveContainer" containerID="c94b9a5c392af22b30c9ee4ca94dc783b9e4ff94402bfebed0160969227a35fa"
Sep 09 21:18:00.398780 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:00.397595    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"]
Sep 09 21:18:00.398859 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:00.398802    2573 scope.go:117] "RemoveContainer" containerID="6b2899cf034aae8089bf067cf27ca4106d8e56e5a19068b244b77621ee4b9072"
Sep 09 21:18:00.401721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:00.401696    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-rbzv2"]
Sep 09 21:18:02.286972 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:02.286942    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" path="/var/lib/kubelet/pods/f9fe733a-eb68-42c6-a9f4-38bff12e50c6/volumes"
Sep 09 21:18:09.172946 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.172912    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"]
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173192    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="storage-initializer"
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173203    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="storage-initializer"
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173212    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container"
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173218    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container"
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173235    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kube-rbac-proxy"
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173241    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kube-rbac-proxy"
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173293    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kube-rbac-proxy"
Sep 09 21:18:09.173385 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.173300    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fe733a-eb68-42c6-a9f4-38bff12e50c6" containerName="kserve-container"
Sep 09 21:18:09.175903 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.175886    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.178505 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.178482    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 09 21:18:09.178505 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.178497    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 09 21:18:09.178726 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.178491    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tlmnz\""
Sep 09 21:18:09.179194 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.179177    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:18:09.179283 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.179238    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:18:09.187636 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.187616    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"]
Sep 09 21:18:09.324163 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.324130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8eff586-bb08-4b01-a557-6a759b62532e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.324163 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.324164    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8eff586-bb08-4b01-a557-6a759b62532e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.324372 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.324226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnxv\" (UniqueName: \"kubernetes.io/projected/c8eff586-bb08-4b01-a557-6a759b62532e-kube-api-access-gnnxv\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.425212 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.425135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnxv\" (UniqueName: \"kubernetes.io/projected/c8eff586-bb08-4b01-a557-6a759b62532e-kube-api-access-gnnxv\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.425364 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.425221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8eff586-bb08-4b01-a557-6a759b62532e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.425364 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.425251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8eff586-bb08-4b01-a557-6a759b62532e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.425960 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.425938    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8eff586-bb08-4b01-a557-6a759b62532e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.427820 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.427801    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8eff586-bb08-4b01-a557-6a759b62532e-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.434699 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.434675    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnxv\" (UniqueName: \"kubernetes.io/projected/c8eff586-bb08-4b01-a557-6a759b62532e-kube-api-access-gnnxv\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7bscb\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.485453 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.485421    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:09.757085 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.757059    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"]
Sep 09 21:18:09.758817 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:18:09.758786    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8eff586_bb08_4b01_a557_6a759b62532e.slice/crio-4e9f51ecf0a60442cea425dfdea0bdd32cfea7ccd59aaeb668db429e7f0fad1c WatchSource:0}: Error finding container 4e9f51ecf0a60442cea425dfdea0bdd32cfea7ccd59aaeb668db429e7f0fad1c: Status 404 returned error can't find the container with id 4e9f51ecf0a60442cea425dfdea0bdd32cfea7ccd59aaeb668db429e7f0fad1c
Sep 09 21:18:09.760864 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:09.760843    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:18:10.419831 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:10.419798    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" event={"ID":"c8eff586-bb08-4b01-a557-6a759b62532e","Type":"ContainerStarted","Data":"4e9f51ecf0a60442cea425dfdea0bdd32cfea7ccd59aaeb668db429e7f0fad1c"}
Sep 09 21:18:11.425804 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:11.425761    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" event={"ID":"c8eff586-bb08-4b01-a557-6a759b62532e","Type":"ContainerStarted","Data":"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"}
Sep 09 21:18:11.426193 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:11.425833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" event={"ID":"c8eff586-bb08-4b01-a557-6a759b62532e","Type":"ContainerStarted","Data":"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"}
Sep 09 21:18:11.426193 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:11.425866    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:11.426193 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:11.425898    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:11.427716 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:11.427692    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:18:11.446656 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:11.446606    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" podStartSLOduration=1.366894032 podStartE2EDuration="2.446592799s" totalImagesPullingTime="1.079698767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:11.445044283 +0000 UTC m=+1563.655588068" watchObservedRunningTime="2026-09-09 21:18:11.446592799 +0000 UTC m=+1563.657136583"
Sep 09 21:18:18.438206 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:18:18.438179    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:19:54.308106 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.308079    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-7bscb_c8eff586-bb08-4b01-a557-6a759b62532e/kserve-container/0.log"
Sep 09 21:19:54.550166 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.550130    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"]
Sep 09 21:19:54.550406 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.550359    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kserve-container" containerID="cri-o://8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a" gracePeriod=30
Sep 09 21:19:54.550487 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.550410    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kube-rbac-proxy" containerID="cri-o://77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf" gracePeriod=30
Sep 09 21:19:54.794164 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.794136    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:19:54.803648 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.803621    2573 generic.go:356] "Generic (PLEG): container finished" podID="c8eff586-bb08-4b01-a557-6a759b62532e" containerID="77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf" exitCode=2
Sep 09 21:19:54.803648 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.803644    2573 generic.go:356] "Generic (PLEG): container finished" podID="c8eff586-bb08-4b01-a557-6a759b62532e" containerID="8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a" exitCode=2
Sep 09 21:19:54.803846 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.803690    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"
Sep 09 21:19:54.803846 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.803704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" event={"ID":"c8eff586-bb08-4b01-a557-6a759b62532e","Type":"ContainerDied","Data":"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"}
Sep 09 21:19:54.803846 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.803759    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" event={"ID":"c8eff586-bb08-4b01-a557-6a759b62532e","Type":"ContainerDied","Data":"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"}
Sep 09 21:19:54.803846 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.803781    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb" event={"ID":"c8eff586-bb08-4b01-a557-6a759b62532e","Type":"ContainerDied","Data":"4e9f51ecf0a60442cea425dfdea0bdd32cfea7ccd59aaeb668db429e7f0fad1c"}
Sep 09 21:19:54.803846 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.803805    2573 scope.go:117] "RemoveContainer" containerID="77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"
Sep 09 21:19:54.809020 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.809001    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8eff586-bb08-4b01-a557-6a759b62532e-proxy-tls\") pod \"c8eff586-bb08-4b01-a557-6a759b62532e\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") "
Sep 09 21:19:54.809114 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.809036    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnnxv\" (UniqueName: \"kubernetes.io/projected/c8eff586-bb08-4b01-a557-6a759b62532e-kube-api-access-gnnxv\") pod \"c8eff586-bb08-4b01-a557-6a759b62532e\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") "
Sep 09 21:19:54.809114 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.809064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8eff586-bb08-4b01-a557-6a759b62532e-kube-rbac-proxy-sar-config\") pod \"c8eff586-bb08-4b01-a557-6a759b62532e\" (UID: \"c8eff586-bb08-4b01-a557-6a759b62532e\") "
Sep 09 21:19:54.809442 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.809419    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8eff586-bb08-4b01-a557-6a759b62532e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8eff586-bb08-4b01-a557-6a759b62532e" (UID: "c8eff586-bb08-4b01-a557-6a759b62532e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:54.811350 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.811279    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eff586-bb08-4b01-a557-6a759b62532e-kube-api-access-gnnxv" (OuterVolumeSpecName: "kube-api-access-gnnxv") pod "c8eff586-bb08-4b01-a557-6a759b62532e" (UID: "c8eff586-bb08-4b01-a557-6a759b62532e"). InnerVolumeSpecName "kube-api-access-gnnxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:54.811443 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.811379    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8eff586-bb08-4b01-a557-6a759b62532e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8eff586-bb08-4b01-a557-6a759b62532e" (UID: "c8eff586-bb08-4b01-a557-6a759b62532e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:19:54.812061 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.812042    2573 scope.go:117] "RemoveContainer" containerID="8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"
Sep 09 21:19:54.825436 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.825411    2573 scope.go:117] "RemoveContainer" containerID="77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"
Sep 09 21:19:54.825790 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:19:54.825771    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf\": container with ID starting with 77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf not found: ID does not exist" containerID="77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"
Sep 09 21:19:54.825866 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.825802    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"} err="failed to get container status \"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf\": rpc error: code = NotFound desc = could not find container \"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf\": container with ID starting with 77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf not found: ID does not exist"
Sep 09 21:19:54.825866 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.825824    2573 scope.go:117] "RemoveContainer" containerID="8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"
Sep 09 21:19:54.826124 ip-10-0-129-205 kubenswrapper[2573]: E0909 21:19:54.826100    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a\": container with ID starting with 8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a not found: ID does not exist" containerID="8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"
Sep 09 21:19:54.826177 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.826136    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"} err="failed to get container status \"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a\": rpc error: code = NotFound desc = could not find container \"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a\": container with ID starting with 8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a not found: ID does not exist"
Sep 09 21:19:54.826177 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.826159    2573 scope.go:117] "RemoveContainer" containerID="77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"
Sep 09 21:19:54.826424 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.826406    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf"} err="failed to get container status \"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf\": rpc error: code = NotFound desc = could not find container \"77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf\": container with ID starting with 77c9ce4fd1d8a6428b5b75ea1c810a80740d68da72d5a351a72bfd65bd6a39cf not found: ID does not exist"
Sep 09 21:19:54.826467 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.826425    2573 scope.go:117] "RemoveContainer" containerID="8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"
Sep 09 21:19:54.826688 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.826671    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a"} err="failed to get container status \"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a\": rpc error: code = NotFound desc = could not find container \"8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a\": container with ID starting with 8e8a72c91837bb4f3829660b1aa70bc926212392a13838e803a06f06b14a561a not found: ID does not exist"
Sep 09 21:19:54.910248 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.910212    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8eff586-bb08-4b01-a557-6a759b62532e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:19:54.910248 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.910242    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8eff586-bb08-4b01-a557-6a759b62532e-proxy-tls\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:19:54.910248 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:54.910251    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnnxv\" (UniqueName: \"kubernetes.io/projected/c8eff586-bb08-4b01-a557-6a759b62532e-kube-api-access-gnnxv\") on node \"ip-10-0-129-205.ec2.internal\" DevicePath \"\""
Sep 09 21:19:55.124960 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:55.124931    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"]
Sep 09 21:19:55.129193 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:55.129171    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7bscb"]
Sep 09 21:19:56.288879 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:19:56.288845    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" path="/var/lib/kubelet/pods/c8eff586-bb08-4b01-a557-6a759b62532e/volumes"
Sep 09 21:20:31.067891 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:31.067813    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c45vx_1cd6adce-d09e-42db-8a98-edca9cfb2c26/global-pull-secret-syncer/0.log"
Sep 09 21:20:31.273881 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:31.273844    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l8c65_d4e9a020-a8af-4c11-8215-5fe31de5cd29/konnectivity-agent/0.log"
Sep 09 21:20:31.296141 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:31.296116    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-205.ec2.internal_c5e0ae4901bfc17af41e47ecf4fc10e2/haproxy/0.log"
Sep 09 21:20:36.398070 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:36.398042    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xvwmr_c9ee83c4-5d84-4f37-80bb-2d28a97288d4/node-exporter/0.log"
Sep 09 21:20:36.429926 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:36.429899    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xvwmr_c9ee83c4-5d84-4f37-80bb-2d28a97288d4/kube-rbac-proxy/0.log"
Sep 09 21:20:36.455395 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:36.455368    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xvwmr_c9ee83c4-5d84-4f37-80bb-2d28a97288d4/init-textfile/0.log"
Sep 09 21:20:39.176328 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.176296    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"]
Sep 09 21:20:39.176721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.176627    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kserve-container"
Sep 09 21:20:39.176721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.176639    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kserve-container"
Sep 09 21:20:39.176721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.176646    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kube-rbac-proxy"
Sep 09 21:20:39.176721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.176651    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kube-rbac-proxy"
Sep 09 21:20:39.176721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.176701    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kube-rbac-proxy"
Sep 09 21:20:39.176721 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.176709    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8eff586-bb08-4b01-a557-6a759b62532e" containerName="kserve-container"
Sep 09 21:20:39.180523 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.180506    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.184591 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.184545    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wltvn\"/\"openshift-service-ca.crt\""
Sep 09 21:20:39.184727 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.184608    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wltvn\"/\"default-dockercfg-rjrx7\""
Sep 09 21:20:39.186019 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.186003    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wltvn\"/\"kube-root-ca.crt\""
Sep 09 21:20:39.192376 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.192353    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"]
Sep 09 21:20:39.253956 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.253909    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-sys\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.253956 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.253952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-proc\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.254208 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.253979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-podres\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.254208 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.254024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljc5x\" (UniqueName: \"kubernetes.io/projected/b509d1fd-0654-49da-9542-1423a380188e-kube-api-access-ljc5x\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.254208 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.254048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-lib-modules\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.354845 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.354813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-proc\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.354998 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.354857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-podres\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.354998 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.354882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljc5x\" (UniqueName: \"kubernetes.io/projected/b509d1fd-0654-49da-9542-1423a380188e-kube-api-access-ljc5x\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.354998 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.354912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-lib-modules\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.354998 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.354927    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-proc\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.355251 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.354998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-sys\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.355251 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.355068    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-sys\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.355251 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.355085    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-podres\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.355251 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.355182    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b509d1fd-0654-49da-9542-1423a380188e-lib-modules\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.376729 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.376704    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljc5x\" (UniqueName: \"kubernetes.io/projected/b509d1fd-0654-49da-9542-1423a380188e-kube-api-access-ljc5x\") pod \"perf-node-gather-daemonset-pskhh\" (UID: \"b509d1fd-0654-49da-9542-1423a380188e\") " pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.490546 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.490456    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.764854 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.764774    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"]
Sep 09 21:20:39.767998 ip-10-0-129-205 kubenswrapper[2573]: W0909 21:20:39.767971    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb509d1fd_0654_49da_9542_1423a380188e.slice/crio-c73f25123d5361bd2ca2b08a8eb36613c5a2ecfa3eec5150a21f5ac69d1f1d70 WatchSource:0}: Error finding container c73f25123d5361bd2ca2b08a8eb36613c5a2ecfa3eec5150a21f5ac69d1f1d70: Status 404 returned error can't find the container with id c73f25123d5361bd2ca2b08a8eb36613c5a2ecfa3eec5150a21f5ac69d1f1d70
Sep 09 21:20:39.972626 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.972582    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh" event={"ID":"b509d1fd-0654-49da-9542-1423a380188e","Type":"ContainerStarted","Data":"c52b5f38583e18bf7c0e9a7545467ecd9f8629d835be9ee7b21c98075e093630"}
Sep 09 21:20:39.972806 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.972639    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh" event={"ID":"b509d1fd-0654-49da-9542-1423a380188e","Type":"ContainerStarted","Data":"c73f25123d5361bd2ca2b08a8eb36613c5a2ecfa3eec5150a21f5ac69d1f1d70"}
Sep 09 21:20:39.972806 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.972710    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:39.993604 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:39.993536    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh" podStartSLOduration=0.993519325 podStartE2EDuration="993.519325ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:39.992577438 +0000 UTC m=+1712.203121214" watchObservedRunningTime="2026-09-09 21:20:39.993519325 +0000 UTC m=+1712.204063109"
Sep 09 21:20:41.373010 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:41.372986    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7fppx_dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5/dns/0.log"
Sep 09 21:20:41.410093 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:41.410067    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7fppx_dc3fa8e7-c21c-43fa-b3e0-adad1ae499b5/kube-rbac-proxy/0.log"
Sep 09 21:20:41.625020 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:41.624941    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dq49d_796b5cc7-bfa3-4f0e-ba87-73d14a885180/dns-node-resolver/0.log"
Sep 09 21:20:42.211136 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:42.211105    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xtcgf_9cfac31b-51fd-4662-8938-4f217ab90c2d/node-ca/0.log"
Sep 09 21:20:43.438442 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:43.438412    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-scgvd_c7177e33-d467-4ed6-bbf5-9687e57c09e4/serve-healthcheck-canary/0.log"
Sep 09 21:20:43.852032 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:43.852006    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-2qt7h_27d62ac1-f632-4428-adf5-25c91d1b8364/insights-operator/0.log"
Sep 09 21:20:43.852213 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:43.852143    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-2qt7h_27d62ac1-f632-4428-adf5-25c91d1b8364/insights-operator/1.log"
Sep 09 21:20:43.878293 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:43.878266    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6hbfv_376174b6-5720-4e8d-859a-9817ebad36b6/kube-rbac-proxy/0.log"
Sep 09 21:20:43.903506 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:43.903471    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6hbfv_376174b6-5720-4e8d-859a-9817ebad36b6/exporter/0.log"
Sep 09 21:20:43.927277 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:43.927250    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6hbfv_376174b6-5720-4e8d-859a-9817ebad36b6/extractor/0.log"
Sep 09 21:20:45.985585 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:45.985533    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wltvn/perf-node-gather-daemonset-pskhh"
Sep 09 21:20:46.484102 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:46.484055    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-8tfg9_759f0e7b-4a99-4da5-934d-dc114a273e9c/manager/0.log"
Sep 09 21:20:46.508911 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:46.508888    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-2vwfl_7bdb300b-1585-4468-912b-57d63e47cdef/server/0.log"
Sep 09 21:20:52.457754 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:52.457694    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-dgrqc_b3434296-b3eb-48ef-b19f-95ce0c894107/kube-storage-version-migrator-operator/1.log"
Sep 09 21:20:52.458546 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:52.458522    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-dgrqc_b3434296-b3eb-48ef-b19f-95ce0c894107/kube-storage-version-migrator-operator/0.log"
Sep 09 21:20:54.346157 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.346131    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q5fd2_f4b4698e-d000-47db-9b23-b583509bcf5f/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:20:54.384338 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.384308    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q5fd2_f4b4698e-d000-47db-9b23-b583509bcf5f/egress-router-binary-copy/0.log"
Sep 09 21:20:54.426009 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.425982    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q5fd2_f4b4698e-d000-47db-9b23-b583509bcf5f/cni-plugins/0.log"
Sep 09 21:20:54.489049 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.489019    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q5fd2_f4b4698e-d000-47db-9b23-b583509bcf5f/bond-cni-plugin/0.log"
Sep 09 21:20:54.536149 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.536109    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q5fd2_f4b4698e-d000-47db-9b23-b583509bcf5f/routeoverride-cni/0.log"
Sep 09 21:20:54.589967 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.589910    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q5fd2_f4b4698e-d000-47db-9b23-b583509bcf5f/whereabouts-cni-bincopy/0.log"
Sep 09 21:20:54.636475 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.636400    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q5fd2_f4b4698e-d000-47db-9b23-b583509bcf5f/whereabouts-cni/0.log"
Sep 09 21:20:54.679324 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.679297    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g7mfh_94f11843-207d-4a2a-89e0-e997b588d1cd/kube-multus/0.log"
Sep 09 21:20:54.918140 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.918063    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xr4nz_6d4bd7ab-dd8f-45d6-a6bc-23642e40c927/network-metrics-daemon/0.log"
Sep 09 21:20:54.995353 ip-10-0-129-205 kubenswrapper[2573]: I0909 21:20:54.995327    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xr4nz_6d4bd7ab-dd8f-45d6-a6bc-23642e40c927/kube-rbac-proxy/0.log"